A Self-Pruning Transformer: Extreme KV-Cache Compression w/Universal Attention (arxiv.org)

Researchers have introduced a novel architecture called Universal Attention, which addresses the significant challenge of large KV-cache sizes in modern large language models (LLMs). Traditional decay mechanisms used in attention layers have limited expressivity, resulting in inefficient pruning methods. Universal Attention presents a comprehensive framework that incorporates adaptive pruning criteria, enabling the removal of tokens that contribute least to computation while maintaining the effectiveness of RoPE embeddings and Softmax attention. This approach allows for enhanced efficiency in deployment and performance improvements in downstream tasks. The implications of Universal Attention for the AI/ML community are substantial, as it achieves state-of-the-art compression ratios—up to 10 times on various datasets and an impressive 25 times with long-context inputs at a length of 16,000 tokens. This level of compression not only facilitates more efficient model deployment but also improves task performance compared to existing methods. By advancing the capabilities of attention mechanisms, Universal Attention could redefine expectations for scalability and performance in LLM applications, paving the way for better utilization of resources in AI systems.