ZAYA1-8B: Frontier intelligence density via 0.7B active MoE trained on AMD (www.zyphra.com)

Zyphra has announced the launch of ZAYA1-8B, a groundbreaking mixture of experts (MoE) model specifically trained on an AMD Instinct™ MI300 stack. This model distinguishes itself by delivering high intelligence density with under 1 billion active parameters, outperforming significantly larger models in complex reasoning, mathematics, and coding tasks. ZAYA1-8B's innovative architecture incorporates features such as Compressed Convolutional Attention (CCA) and a novel router for expert selection, which enhance efficiency and performance while maintaining competitive outputs against leading open-weight models like Mistral-Small-4-119B and Claude 4.5 Sonnet. The significance of ZAYA1-8B lies in its potential to redefine standards for model performance given its compact size and advanced training methodologies, including a novel test-time compute strategy dubbed Markovian RSA. This technique allows the model to perform complex reasoning tasks by efficiently managing context and computation during inference. By integrating these advancements into its training and post-training pipeline, Zyphra aims to demonstrate that smaller models can rival and even exceed the capabilities of larger counterparts, thereby encouraging further innovation within the AI/ML community. The model is now available as a serverless endpoint on Zyphra Cloud, advancing access to cutting-edge AI technology.