Running Gemma 4 26B at 5 tokens/SEC on a 13-year-old Xeon with no GPU (www.neomindlabs.com)

A tech enthusiast has successfully run Google's Gemma 4, a 26-billion-parameter mixture-of-experts language model, on a remarkably outdated hardware setup—a 13-year-old HP StoreVirtual storage box powered by dual Ivy Bridge Xeon processors and lacking a GPU. The model operates at approximately five tokens per second, demonstrating that modern AI models can still execute on legacy hardware when appropriately optimized. This endeavor highlights significant engineering ingenuity as it employed techniques like speculative decoding and CPU-aware routing, tailoring the implementation to work around the limitations of older instruction sets. This achievement is particularly noteworthy for the AI/ML community as it challenges the prevailing notion that high computational power is mandatory for utilizing state-of-the-art models. By adapting Gemma 4 for a CPU-only environment on outdated components, it provides a practical solution for those with limited resources, encouraging further exploration into leveraging existing technologies for AI tasks. The project also emphasizes the importance of understanding model architectures deeply enough to troubleshoot and optimize them, ultimately advocating for a shift from reliance on paid services to more innovative, cost-efficient solutions.