Speculation Is All You Need (modal.com)

This week, a significant advancement in AI model architecture was announced with the release of a state-of-the-art speculative decoder for the Qwen 3.5 model series. Developed in collaboration with Z Lab and SGLang, this new DFlash speculator boosts performance across several models, achieving speedups of 5-20% on various workloads. For example, the Qwen 3.5 122B-A10B model now processes tokens at over 1000 tokens per second compared to just 250 tokens per second without speculation, demonstrating a major leap in inference efficiency. Speculative decoding transforms the decoding phase of large language models (LLMs) by enabling parallel processing of tokens, significantly accelerating output generation. This technique not only facilitates faster inference but also improves acceptance lengths for tasks with extensive context, such as software engineering. As speculative decoding becomes pivotal for achieving high-performance AI applications, it stands out compared to traditional kernel optimizations. The release emphasizes that by harnessing more data and compute resources, models can yield unprecedented speed enhancements, establishing speculative decoding as a critical frontier in AI and ML development.