(Tech Report) DeepSeek-v4.1-Flash: Pushing Limits of KV Cache Compression [pdf] (huggingface.co)

DeepSeek-AI has unveiled its latest innovation, DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model designed to enhance efficiency in managing long-context computations. With a notable backbone of 552 billion parameters, it supports contexts of up to one million tokens. The model leverages a Causal Encoder-Decoder (CED) architecture that significantly improves cost efficiency by activating only 8 billion parameters during prefill and 16 billion during decoding. This design leads to a dramatic reduction in KV cache footprint—approximately one-fourth of its predecessor, DeepSeek-V4-Flash, and one-eighth for persistent KV cache storage. The significance of DeepSeek-V4.1-Flash lies in its optimization of KV cache management, a critical area for deployment in long-horizon AI applications. By implementing CompressedSparseAttention2 (CSA2) for cross-layer KV cache reuse and introducing SWABoundedReplay for efficient state reconstruction, the model alleviates computational, storage, and bandwidth bottlenecks that have hampered previous implementations. These advancements not only lower deployment costs but also enhance performance across diverse text-based and multimodal applications, marking a substantial step forward in AI efficiency and scalability. Model checkpoints are readily accessible on Hugging Face for further exploration by the AI/ML community.