Accelerating Gemma 4: faster inference with multi-token prediction drafters (blog.google)

Gemma AI has announced the release of Multi-Token Prediction (MTP) drafters for its Gemma 4 model, enhancing inference speed by up to three times without compromising output quality. This innovative approach utilizes a speculative decoding architecture, allowing a lightweight drafter to predict multiple future tokens in parallel while the heavier target model processes them, addressing a major latency bottleneck in standard large language model inference. The implications for the AI/ML community are significant, particularly for developers aiming to improve application responsiveness. By integrating MTP drafters, users can reduce latency in real-time applications, enhance offline coding capabilities, and optimize performance on edge devices while preserving battery life. With built-in hardware-specific optimizations, such as efficient KV caching and clustering techniques, MTP drafters promise to optimize resource utilization on consumer-grade hardware, making advanced AI capabilities more accessible and efficient for a wide range of applications.