Linear Attention, Visualized (snowchord.com)

In July 2026, Moonshot unveiled Kimi K3, a groundbreaking 2T-parameter model featuring a 1M-token context window and utilizing an innovative linear-attention mechanism known as Kimi Delta Attention (KDA). Only trailing Claude Fable 5 and GPT-5.6 Sol in performance evaluations, K3 represents a significant advancement in model efficiency by addressing the computational bottlenecks of traditional self-attention, which is quadratic in time complexity. The model's design aligns with NVIDIA's concurrent developments in Nemotron 3, demonstrating a trend toward more efficient attention mechanisms in AI systems. KDA employs a linear complexity approach by recalibrating how queries interact with keys and values, thus avoiding the storage of a large score matrix. This transition from quadratic to linear operations not only enhances processing speed but also allows for real-time applications with longer context windows. Furthermore, the model incorporates innovative features like the chunkwise algorithm, which optimally balances computation across tokens while maintaining energy efficiency on GPU architectures. With K3's advancements, it is poised to influence future AI architectures, making it a pivotal development in the AI/ML community as they seek to scale model capabilities without compromising on efficiency.