Ministral 3 – pruning via Cascade Distillation (arxiv.org)

Ministral has announced the launch of the Ministral 3 series, a collection of parameter-efficient language models tailored for applications with constrained computing and memory resources. The series is available in three sizes—3B, 8B, and 14B parameters—each offering three specialized variants: a general-purpose pretrained model, an instruction-finetuned model, and a reasoning model optimized for complex problem-solving. Significantly, the models incorporate image understanding capabilities and are released under the Apache 2.0 license, promoting accessibility in the AI community. The creation of Ministral 3 employs an innovative technique called Cascade Distillation, which involves iterative pruning and continued training through distillation. This method not only enhances the models' efficiency but also their performance in delivering reliable language processing in resource-limited scenarios. The significance of this advancement lies in its potential to democratize access to powerful AI tools, enabling smaller organizations and individuals to leverage advanced language understanding without necessitating extensive computational resources.