DeepSeek V4's indexer OOMs at 65K context. We got it to 1M in 6G (arxiv.org)

Researchers have unveiled StreamIndex, an innovative solution addressing the limitations of previous versions of the DeepSeek framework, specifically V3.2 and V4, which faced out-of-memory (OOM) issues at high sequence lengths due to their compressed sparse attention (CSA) mechanism. The challenge stemmed from the massive intermediate FP32 score tensor generated during the indexing process, which could not be accommodated by conventional GPU memory. StreamIndex employs a chunked partition-merge approach to perform top-k selection without materializing the full score tensor, enabling it to efficiently handle sequences up to 1,048,576 tokens on a single NVIDIA H200, significantly extending the usable context length by 32 times. This development has vital implications for the AI/ML community, particularly in advancing natural language processing and large-scale machine learning tasks. By overcoming memory constraints, StreamIndex not only enhances the scalability of attention mechanisms but also maintains high accuracy, achieving near-perfect recall across multiple design experiments. These advancements facilitate the processing of larger datasets and complex models, positioning StreamIndex as a crucial tool for researchers and developers working with cutting-edge AI applications.