DiffusionGemma: Discrete diffusion in a large language model (idlemachines.co.uk)

DeepMind has launched DiffusionGemma, a new model that shifts from the traditional left-to-right autoregression to a discrete diffusion approach, allowing for parallel processing of entire token sequences. This innovative method achieves an impressive throughput of over 1000 tokens per second on a single H100 GPU, outperforming previous autoregressive models by up to four times in size. While it currently doesn’t quite match the capabilities of the flagship Gemma-4, it is rapidly advancing in performance and efficiency. This move is significant for the AI/ML community as it challenges conventional training assumptions, particularly around the necessity of end-to-end training in transformers. The model's architecture, along with methodologies from Sakana AI, enables the treatment of each block in the network as an independent diffusion denoiser, eliminating many traditional training bottlenecks related to memory and communication. Additionally, innovations in model design such as NVIDIA's open Nemotron 3 family and the introduction of hybrid layers promise to fundamentally change how we approach large language models, reducing inference costs and improving scalability in millisecond-time contexts.