Optimizing MiniMax M3 Sparse Attention on Nvidia Blackwell (fireworks.ai)

The Fireworks AI Performance team has announced a new kernel for MiniMax M3 sparse attention optimized for Nvidia's Blackwell architecture, marking a significant leap in efficiency for long-context inference tasks. This kernel employs a KV-stationary execution path, allowing the algorithm to effectively manage irregular memory access patterns that typically hinder performance. By optimizing the selection of the top 16 most relevant 128-token key-value (KV) blocks and focusing on minimizing memory traffic while enhancing load balancing across Streaming Multiprocessors (SMs), the new implementation achieves remarkable throughput improvements—peaking at nearly 980 TFLOP/s, outperforming traditional Q-outer approaches by 1.9-2.4 times. The kernel’s design allows for blockwise sparse attention over grouped-query attention (GQA) while maintaining efficient memory access, crucial for scaling model progress in AI/ML. It creatively addresses the trade-offs between different kernel structures by leveraging the unique memory bandwidth characteristics of L2 and High Bandwidth Memory (HBM), leading to a more efficient processing regime at higher utilization rates. This advancement not only enhances the speed and efficiency of attention mechanisms but also underscores the continuing evolution of AI architectures, potentially influencing future developments in efficient inference methodologies across various applications.