Fable 5 pushed Gemma 4 to 255 tok/s on WebGPU (xcancel.com)

Fable 5 has showcased exceptional performance by pushing the Gemma 4 model to an unprecedented 255 tokens per second (tok/s) using WebGPU before its shut down. Initially, the model reached 84 tok/s with some limitations on further optimizations. However, after adjustments from Anthropic to rollback certain invisible safeguards in LLM development, the performance surged to 255 tok/s. This leap underscores the potential of agentic kernel optimization for on-device inference, paving the way for more efficient and responsive AI applications. The significance of this achievement lies in its demonstration of high-performance model inference directly in web environments, which could greatly enhance user experiences in AI-driven applications. The availability of the demo and kernels enables developers and researchers to experiment with these optimizations locally in their browsers, thereby fostering community engagement and further advancements in machine learning. The rapid increases in inference speed show promise for future applications and could generate interest in edge computing solutions that leverage similar optimizations.