Recursive Sparse Attention: Million-Token Extrapolation at Logarithmic Cost (haoailab.com)

🤖 AI Summary
RecSA (Recursive Sparse Attention) introduces a revolutionary approach to handling long-context processing in AI models, allowing them to extrapolate from 32K tokens to 2 million tokens with significantly reduced computational costs. Traditional dense attention mechanisms struggle with scalability as context length increases, leading to slower performance and exceeding memory limits. RecSA circumvents these issues by building a tree structure over the key-value cache, enabling efficient level-by-level query search with a selection cost of O(log L), while retaining the crucial relative distances learned during training. This advancement allows RecSA, initialized from Qwen3-4B, to outperform existing models by a considerable margin—achieving scores of 60.5 at 1 million tokens compared to the 40.0 of the best baseline. The implications of RecSA are profound for the AI/ML community, particularly in applications that demand the processing of extensive data sets, such as document question answering and code generation. By maintaining high performance even at longer contexts, RecSA stands to enhance the effectiveness of AI models in real-world scenarios. Key technical innovations include the use of candidate-local RoPE (Rotary Positional Encoding) to preserve relative distance integrity and a novel pooling mechanism that effectively summarizes key-value nodes without information loss. As a result, RecSA not only accelerates processing speed—three times faster than dense methods for inference at 524K tokens—but also maintains model accuracy across extended token sequences, thereby paving the way for new possibilities in AI applications.
Loading comments...
loading comments...