VeriCache: Turning Lossy KV Cache into Lossless LLM Inference (arxiv.org)

Researchers have introduced VeriCache, a groundbreaking inference framework designed to overcome the bottleneck posed by large KV caches while serving large language models (LLMs) with extended context lengths. Traditional methods of compressing KV caches, like token dropping and quantization, often lead to lossy outputs, causing significant errors in applications like code generation. VeriCache addresses this challenge by enabling lossless inference while maintaining the high throughput typical of existing compression methods. It drafts tokens using a compressed KV cache and verifies them against the full KV cache, cleverly managing memory by paralleling operations and extending the drafting horizon to reduce verification overhead. The significance of VeriCache lies in its capacity to achieve up to 4X greater throughput compared to full-KV inference while guaranteeing identical outputs, enhancing the performance and reliability of LLMs. This approach not only supports a diverse set of compression techniques through a standard interface but also integrates well with traditional speculative decoding methods. With implications for both long-context decoding and remote prefix caching, VeriCache could pave the way for more efficient and accurate AI applications, pushing forward the capabilities of machine learning in real-world scenarios.