DiffusionGemma: 1k tok/s on an H100, 43 tok/s on a Mac (astgl.com)

Google's new model, DiffusionGemma, boasts a capacity of generating over 1,000 tokens per second on high-performance NVIDIA H100 GPUs, contrasting sharply with the typical output of local models, which ranges from 30 to 100 tokens per second. However, a recent benchmark run on a Mac Studio revealed that DiffusionGemma generates only 43 tokens per second, lagging behind the traditional autoregressive Gemma 4, which produces 61 tokens per second. This discrepancy highlights that while diffusion models have exciting potential, their performance is highly dependent on hardware capabilities, particularly when it comes to memory bandwidth. The results uncover critical insights about the mechanics of diffusion models versus autoregressive models. DiffusionGemma's method of refining 256 tokens in parallel offers speed advantages in data centers but introduces latency on consumer-grade machines, as evidenced by a significantly higher time-to-first-token metric. The Mac's architecture favors autoregressive processing, which reuses past computations and thus operates more efficiently for local inference. This benchmark underscores the importance of testing models in real-world settings rather than relying solely on marketing claims, emphasizing that hardware compatibility plays a crucial role in AI model performance. For those interested in experimenting with these findings, a reusable benchmarking harness is available on GitHub.