IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse (github.com)

Researchers have announced IndexCache, a novel approach that accelerates sparse attention in deep learning models by reusing indices across layers, leading to reductions of up to 75% in index computations. This innovation results in significant performance enhancements, achieving a 1.82 times speedup in prefill operations and a 1.48 times speedup during decoding, all while minimally impacting output quality. The method targets Deep Sparse Attention (DSA) mechanisms, which often face inefficiency due to independent indexer computations at each layer. The significance of IndexCache lies in its potential to dramatically enhance the efficiency of language models, particularly for those processing extensive context lengths. By partitioning layers into Full and Shared categories—where Full layers maintain their individual indexer and Shared layers utilize cached indices from adjacent Full layers—IndexCache streamlines the indexing process without additional GPU memory requirements. The research also provides two implementation strategies: a training-free option that optimizes indexer selection based on calibration data, and a training-aware method that fine-tunes retained indexers for broader coverage across layers. This innovation promises not only to improve processing times across various benchmarks but also supports existing DSA architectures, marking a significant advancement for the AI/ML community.