Gemma 3 1B at 114 tokens per second on a laptop CPU (knl.co.in)

A researcher successfully optimized Google's Gemma 3 1B model to achieve an impressive generation speed of 114 tokens per second on a standard laptop CPU by employing a ternary weight system and a specialized inference engine. By retraining the model's decoder to use only three weight values (-1, 0, +1) and implementing a unique kernel that reduces byte consumption, the performance outstrips the previous methods using llama.cpp, which only managed 30 tokens per second. However, this speed increase comes at a cost: the quality of generated text is about 14% worse on standard benchmarks compared to the original model. The technical breakthrough lies in the effective combination of reduced weight representation, leveraging a 1.6-bit kernel, and an innovative output layer strategy called the "sketch head," which efficiently narrows down scoring to only the most promising word candidates. While this advancement allows faster batch processing and ensures the model can handle multiple user requests, it exposes limitations in quality and context retention. As the model pushes against hardware boundaries, further enhancements and training strategies are required to close the quality gap and catalyze even greater performance.