KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache (arxiv.org)

Researchers have introduced KVComp, a high-performance, lossy compression framework specifically designed for managing the key-value (KV) cache in large language models (LLMs) during long-context inference. The KV cache, essential for storing intermediate computations in transformer-based LLMs, can balloon to several gigabytes when processing extensive sequences and batch sizes, posing significant memory challenges. KVComp addresses this by employing novel compression algorithms tailored to the unique data patterns of KV caches, enabling substantial memory savings without compromising model accuracy. KVComp stands out by balancing compression efficiency with computational speed, achieving up to 83% better memory reduction compared to existing methods on average, while maintaining negligible or no loss in model performance. Notably, it reduces decompression overhead to the extent that it can accelerate certain matrix-vector multiplications, outperforming traditional GPU-accelerated attention kernels like those based on cuBLAS. This synergy of algorithmic innovation and system-level optimization makes KVComp highly suitable for both latency-sensitive and throughput-oriented LLM inference tasks, expanding the feasible context length and batch size in real-world applications. For the AI/ML community, KVComp offers a breakthrough in scaling transformer models efficiently, enabling more practical deployment of long-text generation and extended dialogue systems. Its approach exemplifies the growing importance of hardware-aware and data-specific compression techniques in overcoming memory bottlenecks, which is critical as LLMs continue to grow in size and complexity.