Show HN: Gemma 4 26B at 100 tok/s on a Mac, using the Neural Engine and 2GB RAM (github.com)

Gemma 4 26B-A4B has demonstrated impressive performance by achieving up to 100 tokens per second on a 16 GB Mac using the Apple Neural Engine (ANE) while consuming less than 2 GB of RAM. This model utilizes a combination of prefill processing by the ANE and speculative decoding on the GPU, allowing for efficient memory management and rapid text generation. Specifically, tests showed that while traditional methods yielded slower generation rates, Gemma's architecture keeps the GPU busy around 93-95% of the time, optimizing processing speed and resource use. This announcement is significant for the AI/ML community as it highlights the potential of utilizing low-power hardware, like Macs with ANE, for running large language models effectively. The innovative approach to expert routing and memory caching, where a 13 GB model runs in under 1 GB of process memory, paves the way for more efficient deployments of complex models in practical applications. Furthermore, integrating Apple’s private frameworks hints at exciting possibilities for model optimization, although it also poses risks regarding future compatibility with updates to macOS. Overall, Gemma 4 exemplifies the evolution of AI capabilities harnessed on consumer-grade hardware, making advanced models more accessible.