Qwen3-Next series represents our next-generation foundation models (huggingface.co)

The Qwen3-Next series introduces cutting-edge foundation models designed to dramatically improve efficiency and context handling in large language models. Central to this breakthrough are architectural innovations such as Hybrid Attention—a blend of Gated DeltaNet and Gated Attention—that enhances context modeling while reducing computational load. Additionally, the models employ an extreme sparsity Mixture of Experts (MoE) layer with a 1:50 activation ratio, cutting FLOPs per token significantly without compromising capacity. Multi-Token Prediction (MTP) further boosts both pretraining performance and inference speed, combined with novel normalization and stabilization techniques for robust training. A flagship model, Qwen3-Next-80B-A3B, leverages these innovations to operate with 80 billion parameters but only 3 billion active at a time, achieving over 10 times the inference throughput of its 32-billion-parameter predecessor on sequences longer than 32,000 tokens. Remarkably, it also surpasses Qwen3-32B in downstream task performance while reducing training costs by over 90%. This leap in sparsity-driven efficiency and long-context capability sets a new benchmark for large-scale AI models, promising cost-effective, high-throughput deployment for complex language tasks. Technically, Qwen3-Next models support up to 32K tokens context length, use advanced RoPE positional embeddings with flexible scaling, and feature a configurable multi-head attention system including Grouped Query Attention. This flexibility paired with aggressive sparsity mechanisms underlines their potential for scalable, practical AI applications needing extensive memory and speed without exponential computational expense.