MiMo-V3 is getting a new architecture. HySparse2 (twitter.com)

MiMo-V3 has launched a new architecture, HySparse2, which significantly improves performance by addressing key challenges in handling long-context retrieval in AI/ML scenarios. This architecture introduces notable advancements compared to the previous MiMo-V2.6, boasting a 5.02× reduction in prefill FLOPs, a 4.5× smaller key-value (KV) cache, and enhanced scores on metrics like MRCRv2 and RULER-v2. By optimizing these elements, HySparse2 aims to streamline efficiency in workloads, particularly those involving agentic inference where quick actions yield extensive observations. The innovation centers around a two-level KV sharing system, which includes KV Bridging and KV Reuse. KV Bridging allows full-attention layers within the cross-decoder to generate their key and value pairs from the self-decoder's hidden states, while KV Reuse enables sparse layers to utilize the preceding full-attention layer's KV cache. Additionally, the architecture shifts from block-level to token-level selection and integrates a forced window of recent tokens, enhancing both local and global context processing. This integrated approach not only reduces computational load but also facilitates a smoother retrieval process, marking a significant leap forward for AI systems requiring efficient handling of long-range dependencies.