Sarvamai/Sarvam-105B (huggingface.co)

The Sarvam-105B, a cutting-edge Mixture-of-Experts (MoE) model featuring 10.3 billion active parameters, has been released as an open-source tool under the Apache License. This model excels in complex reasoning tasks, mathematics, and coding, proving its mettle by consistently competing with advanced closed-source models. Notably, it has been optimized to perform exceptionally well in 22 Indian languages, enriching accessibility and usability for users within that context. With a simple interface using Hugging Face, developers can easily implement this model for various applications. Technically, Sarvam-105B employs an MLA-style attention stack with advanced scaling techniques that enhance performance dramatically, allowing for context lengths of up to 65,536 tokens. Its innovative architecture includes features like decoupled QK head dimensions and top-8 routing over 128 experts, boosting representational capacity while managing computational costs. The model's performance on several benchmarks confirms high accuracy across areas like math and coding, making it a significant advancement for the AI/ML community, particularly in addressing multilingual and complex cognitive tasks.