LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization (leapquant.github.io)

LeapQuant is a groundbreaking training-free method that quantizes the recurrent state of linear attention models to 8 bits while maintaining near-lossless model quality, achieving accuracies comparable to traditional FP32 baseline methods. This innovative approach results in significant performance enhancements, including average speedups of 2.05–3.70× at the kernel level and 1.57× for end-to-end inference on popular NVIDIA GPUs like the B200, RTX PRO 6000, and RTX 5090. As linear attention techniques become commonplace in large language models (LLMs), LeapQuant's ability to lower memory traffic by 3.4× and reduce overall memory footprint by up to 56% is particularly significant for the AI/ML community, addressing the persistent memory bandwidth bottleneck associated with decoding processes in these models. Central to LeapQuant's success are three novel training-free techniques that minimize quantization error during the recurrent state updates. It strategically delays quantization by leaps across token windows, capturing high-magnitude outliers as Compensator Tokens, and smoothing the quantization residuals to ensure more balanced quantization. This method not only enhances computational efficiency but also preserves model accuracy across various LLM families, outperforming naive quantization strategies that often result in substantial accuracy loss. By enabling efficient long-context processing without sacrificing quality, LeapQuant is poised to make a lasting impact on the development and deployment of advanced AI models.