Mistral Large 4 on AlphaXiv (www.alphaxiv.org)

Mistral Large 4 (ML4), a groundbreaking multimodal mixture-of-experts model with 52 billion active parameters and a total of 1 trillion parameters, has been introduced as a solution for reasoning and long-horizon agentic tasks. This model achieves competitive scores on various benchmarks, such as 19.7% on FrontierCode 2.0 and 41% on Terminal-Bench 4.0. The architecture incorporates innovative techniques like ragged multi-head latent attention, which reduces context storage requirements, and DeepSeek Sparse Attention (DSA), allowing the model to attend only to selected past tokens. Efficient serving is further enhanced through calibrated NVFP4 quantization, reducing the model's size to 606 GB for deployment on four GB300 GPUs, and speculative decoding which improves interactivity by up to 90%. The significance of Mistral Large 4 for the AI/ML community lies in its ability to balance long-context requirements with practical serving constraints. Its design evolved by focusing not just on language modeling loss but on enhancing agentic performance, illustrating that traditional metrics may not fully capture a model's real-world capabilities. The model's success hinges on effective architecture adjustments, such as restoring full attention while maintaining cost-efficiency through sparse attention methods. This suggests a paradigm shift in architecting AI models focused on real-world applications in software engineering and cybersecurity, making it a pivotal contributor to advancing multimodal AI systems.