Google’s Gemma 4 open AI models use “speculative decoding” to get up to 3x faster (arstechnica.com)

Google has introduced the Gemma 4 open AI models, enhancing their performance with the innovative Multi-Token Prediction (MTP) draft models, which utilize speculative decoding. This approach allows the model to predict future tokens faster than traditional autoregressive methods, which generate one token at a time. By reducing unnecessary computation time typically spent transferring data between system memory and processing units, MTP can potentially triple the speed of token generation. The models are designed to run locally, powered by Google’s TPU architecture, making them more accessible for users who prefer to process data privately without relying on cloud systems. The significance of Gemma 4 lies in its adaptability and user empowerment within the AI/ML community. With a shift to the more permissive Apache 2.0 license, developers have greater freedom to experiment with the models on their own hardware, which could lead to broader innovation and customization. While the underlying technology mirrors Google’s Gemini AI, the lightweight, 74 million parameter drafter models are streamlined for efficiency, significantly accelerating the token prediction process. This advancement addresses the limitations of consumer hardware, making AI tools more attainable and practical for a wider audience, and paving the way for enhanced local AI applications.