Speculative KV coding: losslessly compressing KV cache by up to ~4× (fergusfinn.com)

A new method called Speculative KV Coding has been introduced, enabling lossless compression of key-value (KV) caches used in large language models (LLMs) by up to approximately four times. This technique leverages a predictor model that outputs estimates of the original KV cache, which are then encoded by an arithmetic coder. This approach addresses the growing memory demands associated with longer context sizes in LLMs, providing a means to reduce storage and computational overhead without sacrificing quality. Traditional lossy compression techniques, like TurboQuant, trade off accuracy for size, whereas this innovative method guarantees exact reconstruction of the original cache data, making it a more efficient solution for handling large-scale LLM operations. The significance of this advancement lies in its potential to enhance the efficiency of LLM deployment by significantly lowering memory requirements while maintaining model performance. The employed predictor, typically a quantized version of the original model, allows for easier data management as the arithmetic coder minimizes the compression cost driven by correct KV cache predictions. As demonstrated with the Qwen3 model family, this method shows promising results, achieving compression ratios of over six times when stacked with existing lossy quantization techniques. This advancement not only promises streamlined LLM operations but also opens avenues for future research into optimizing model architectures and distribution management.