0.5B base model trained on 60B tokens rivals Qwen3.5-2B trained on 30T+ tokens (huggingface.co)

The AI community has a new contender in the form of Nenya, a general-purpose 0.5B active, 2.5B total parameter mixture-of-experts (MoE) model that has demonstrated competitive performance against Qwen3.5, which was trained on over 30 trillion tokens. Remarkably, Nenya achieved this with 500 times lower training compute and 3.4 times fewer active FLOPs per token. This advancement highlights the potential for more efficient model designs that maintain high performance, a vital consideration given the growing environmental and economic costs of large-scale AI training. Nenya's architecture features 32 layers, 120 experts, and an innovative attention mechanism combining GDN-2 and MLA types, allowing for efficient routing of expert parameters. With a context length of 16,384 tokens and a vocabulary size of 100,278, this model, built primarily on web data, educational, and coding resources, shows promising accuracy across various benchmarks, notably competing effectively against larger models despite its significantly smaller size. The development of such efficient models may reshape strategies for training and deploying AI applications, encouraging further exploration of mixture-of-experts architectures in the field.