A simple way to compress model, KV cache, flops vias low-rank with zero overhead (jeffreywong20.github.io)

A3, a new post-training low-rank approximation framework, has been proposed to enhance the efficiency of large language models (LLMs) like LLaMA. Unlike previous methods that focus on individual linear layers and introduce runtime overhead, A3 breaks down each Transformer layer into three key components—query/key (QK), output value (OV), and multi-layer perceptron (MLP)—allowing for direct reduction of hidden dimensions while minimizing functional loss. This innovative approach results in a significant compression of model size, key-value cache, and computational workload, all without added inference overhead. For instance, A3 has achieved a perplexity score of 4.69 on WikiText-2 using LLaMA 3.1-70B, substantially outperforming the previous state-of-the-art score of 7.87. The significance of A3 lies in its ability to maintain high model performance while significantly reducing memory and computational costs, which bolsters the practical deployment of large-scale AI systems. By deriving optimal solutions for each component independently and supporting various modern architectures, A3 demonstrates a substantial gain in efficiency across multiple benchmarks. The framework promises a seamless integration into existing inference pipelines, making it a vital development for the AI/ML community seeking to efficiently leverage powerful models in resource-constrained environments.