A general tensor-structured compression scheme for efficient LLMs (www.alphaxiv.org)

A recent development in the AI/ML community, the Tensor Mixture (MixT) compression scheme, addresses the limitations of large language models (LLMs) due to their dense linear transformations. This innovative approach substitutes specific dense linear layers with mixtures of tensor operators that can be executed natively. Importantly, MixT can be broadly applied across various Transformer-based LLMs, making it a versatile tool for enhancing model efficiency without losing significant functionality. The significance of MixT is underscored by its impressive results during evaluations on models like Qwen3-8B and LLaMA2-7B, whereby it managed to retain accuracy within a specific compressible regime. Notably, at the LLaMA2-7B transition boundary, MixT achieved a remarkable reduction in model parameters by 47.5%, along with decreased computational demands—37.1% fewer FLOPs for inference, 52.1% for training, and a 60.4% cut in peak inference memory. This substantial efficiency gain demonstrates MixT's potential for making LLM adaptation and deployment more cost-effective, which could greatly impact the scalability and accessibility of AI technologies.