Xiaomi MiMo-v2.5-Pro Open-Sourced: 1T Parameter Model (huggingface.co)

Xiaomi has announced the open-source release of its MiMo-V2.5-Pro, a cutting-edge Mixture-of-Experts (MoE) language model boasting an impressive 1.02 trillion total parameters with 42 billion active parameters. The model incorporates a hybrid attention architecture and innovative Multi-Token Prediction (MTP) mechanisms, allowing it to handle a colossal context length of up to 1 million tokens. Designed to excel in complex software engineering tasks that require long-horizon reasoning, MiMo-V2.5-Pro effectively sustains intricate interactions involving numerous tool calls while maintaining coherence and response accuracy. This development is significant for the AI/ML community as it pushes the boundaries of long-context reasoning and efficiency in large language models. The hybrid attention approach drastically reduces the storage requirements for key-value (KV) caches, achieving nearly seven times the efficiency without sacrificing performance. Furthermore, the model's quick output generation—thanks to its MTP modules—makes it particularly beneficial for reinforcement learning applications. By employing a sophisticated post-training paradigm that harmonizes supervised fine-tuning with domain-specific reinforcement learning techniques, Xiaomi aims to set a new standard in performance for agentic tasks, encouraging broader experimentation and innovation within the field.