Simplified Sparse Attention via Gist Tokens (arxiv.org)

Researchers have introduced Simplified Sparse Attention (SSA), a novel method for optimizing long-context inference in language models without requiring architectural changes. By utilizing gist tokens during continued pretraining, SSA enables models to concentrate essential information from various context chunks into these tokens. This strategic approach minimizes memory-bandwidth costs by allowing the model to score queries only against gist tokens rather than the full context, leading to significant efficiency improvements. The significance of SSA lies in its ability to enhance performance in tasks such as retrieval-augmented generation, where it outperformed traditional full attention mechanisms by over 5.7 points after pretraining. Additionally, the method extends to a hierarchical version, H-SSA, which achieves log-linear decoding complexity while maintaining high accuracy even at compression ratios up to 32x. By effectively filtering out irrelevant noise and concentrating on query-relevant information, SSA marks a substantial advance in sparse attention techniques, promising greater efficiency and performance for the AI/ML community.