HySparse: A Hybrid Sparse Attention Architecture (arxiv.org)

Researchers have unveiled HySparse, a novel Hybrid Sparse Attention architecture that enhances the efficiency of attention mechanisms in machine learning models. By alternating between full and sparse attention layers, HySparse utilizes the full attention layer as an oracle for token selection and intelligently shares the key-value (KV) caches from the full layers with the sparse layers. This approach addresses two major issues in traditional sparse attention frameworks: the reliance on imprecise proxies for token importance and ineffective use of KV cache resources. HySparse demonstrates significant performance improvements over previous attention models, particularly on large-scale datasets, showing that it can achieve substantial gains in accuracy while reducing KV cache storage by nearly 10x. Evaluated on both 7B dense and 80B mixture-of-experts (MoE) models, HySparse outperformed full attention and hybrid baselines, with only 5 out of 49 layers requiring full attention in the 80B MoE model. This architecture not only simplifies the token selection process but also optimizes memory use, making it a compelling advancement for the AI/ML community focused on enhancing model efficiency and scalability.