Kimi Linear: An Expressive, Efficient Attention Architecture (2025) (arxiv.org)

Researchers have introduced Kimi Linear, a groundbreaking hybrid linear attention architecture that surpasses traditional full attention mechanisms in performance across various contexts, including short and long sequences, as well as reinforcement learning scenarios. At its core, Kimi Linear utilizes Kimi Delta Attention (KDA), an innovative linear attention module that enhances Gated DeltaNet with advanced gating techniques to optimize finite state RNN memory usage. The architecture leverages a specialized Diagonal-Plus-Low-Rank (DPLR) algorithm, significantly reducing computational overhead while maintaining alignment with classical delta rules. The Kimi Linear model, comprising 3 billion activated parameters and 48 billion total parameters, demonstrated remarkable efficiency gains by reducing key-value (KV) cache usage by up to 75% and achieving decoding speeds up to six times faster for contexts of one million tokens. These results suggest that Kimi Linear can serve as a highly efficient alternative to conventional attention mechanisms, especially for applications involving longer input and output sequences. To foster further research, the authors have made the KDA kernel, vLLM implementations, and pre-trained model checkpoints publicly available. This development holds great significance for the AI/ML community, as it could lead to more efficient models that are easier to scale and deploy in real-world applications.