Fusing a 27B ternary LLM's whole decode step into one CUDA kernel (twitter.com)

The recent open-source release of bonsai-turbo, a decode engine that processes the Bonsai 27B model from PrismML, achieves a remarkable performance boost, running 1.76 times faster than the official llama.cpp fork. With speeds of 151 tokens per second for ternary precision and 159 tokens per second for 1-bit precision, it delivers consistent outputs across 32 test prompts without any loss in quality. This enhancement is particularly significant for the AI/ML community as it reflects a breakthrough in optimizing decoding efficiency on GPU architectures, specifically for batch-1 scenarios. The speed improvement stems from addressing GPU overhead limitations rather than traditional math or bandwidth constraints. By fusing the entire per-token processing into a few large operations, bonsai-turbo minimizes overhead usage, compared to the stock path, which performs 3,703 GPU operations per token and incurs about 97% overhead. This innovative approach, facilitated by an internal agent that also drives @runinfrai, positions the Bonsai 27B model as a viable option for local single-user applications such as mobile devices, enhancing user experience significantly. Future developments aim to implement weight pipelining and introduce a speculative drafter, potentially raising speeds to 300 tokens per second.