DeepSeek-V4 KV Cache Explained: Why 1M Context Uses Less VRAM (knightli.com)

DeepSeek-V4 has introduced a groundbreaking change in handling long-context models, particularly regarding how the Key and Value (KV) Cache consumes VRAM during inference. Rather than merely increasing the cache size with context length, DeepSeek-V4 significantly compresses the KV Cache by aggregating multiple historical tokens into fewer entries. In a setup that processes one million tokens, DeepSeek-V4-Pro’s KV Cache consumes about 10% of the memory used by its predecessor, DeepSeek-V3.2, positioning it as a formidable advancement for models dealing with extensive context, such as those used in code navigation, document analysis, and complex agent workflows. This model employs multiple compression techniques, including Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), which allow for efficient information retrieval from long contexts while preserving crucial details for recent tokens through a sliding window mechanism. This approach reduces both the memory footprint and the time to the first token without compromising performance. DeepSeek-V4’s strategy marks a shift from traditional methods that focused on reducing the number of KV heads to optimizing the number of historical token entries, making it a significant step toward practical deployment of million-token contexts in real-world applications.