ZAYA1-8B: An 8B Moe Model with 760M Active Params Matching DeepSeek-R1 on Math (firethering.com)

Zyphra has unveiled ZAYA1-8B, a cutting-edge mixture of experts (MoE) model that features 8.4 billion total parameters, but operates with only 760 million active parameters during inference. This model has demonstrated impressive performance, matching or exceeding benchmarks of major models like DeepSeek-R1 and Claude Sonnet 4.5 in mathematics and maintaining competitiveness with Gemini 2.5 Pro in coding tasks. Significantly, ZAYA1-8B was trained entirely on AMD hardware, specifically the MI300X GPUs, marking a departure from the NVIDIA-dominated landscape in AI infrastructure. This accomplishment not only showcases the viability of AMD for high-performance AI training but also suggests potential cost savings for research labs seeking alternatives to NVIDIA. The underlying engineering hurdles Zyphra overcame are notable; ZAYA1-8B employs a unique Markovian RSA inference method that allows it to reason in manageable chunks while generating multiple parallel traces. This innovation enhances the model's performance as computational resources increase, pushing the boundaries of what's possible with a smaller parameter count. While ZAYA1-8B excels in math and coding, it reveals limitations in multi-step instruction following and tool-use capabilities, making it best suited for specialized applications in scientific and coding contexts. Researchers interested in AI infrastructure, test-time compute methods, and MoE innovations will find this model and its implications particularly valuable.