DashAttention: Differentiable and Adaptable Sparse Hierarchical Attention (arxiv.org)

The recent development of DashAttention introduces a novel approach to hierarchical attention mechanisms, addressing key limitations in existing methods like NSA and InfLLMv2. Traditional hierarchical attention methods rely on a fixed top-k selection of key-value blocks, which limits the adaptability of the model to varying query contexts and obstructs gradient flow between sparse and dense stages. DashAttention innovates by utilizing an adaptive sparse α-entmax transformation, allowing the selection of a variable number of blocks tailored to each specific query. This change enhances the model's differentiability and improves its capability to handle long-context scenarios. Significantly, DashAttention has demonstrated performance on par with full attention models while achieving 75% sparsity, outperforming previous models on the Pareto frontier, particularly in high-sparsity situations. Additionally, its GPU-aware implementation in Triton yields significant efficiency gains, achieving a speedup over FlashAttention-3 during inference. This development marks a substantial advancement for the AI/ML community, particularly for applications requiring effective long-context modeling while maintaining computational efficiency, paving the way for more scalable and adaptable AI systems.