Unlocking Non-Uniform KV Cache for Efficient Multi-Turn LLM Serving (arxiv.org)

A new serving system for Multi-turn Large Language Models (LLMs), named Tangram, has been introduced to tackle the inefficiencies arising from non-uniform Key-Value (KV) caches. These caches are essential for maintaining coherent user interactions, but their linear growth challenges GPU memory and bandwidth, resulting in performance bottlenecks. Tangram leverages non-uniform KV compression to retain vital information while addressing systemic issues such as memory fragmentation and scheduling complexities that typically hinder existing LLM serving systems. Tangram employs three innovative techniques to enhance efficiency: Deterministic Budget Allocation eliminates dynamic scheduling overhead by assigning static memory footprints based on individual head patterns; Head Group Page management clusters attention heads with similar retention needs, thus optimizing memory use; and Ahead-of-Time Load Balancing ensures consistent GPU utilization without introducing runtime delays. Experimental results demonstrate that Tangram can improve throughput by up to 2.6 times compared to current baselines while maintaining model accuracy, marking a significant advancement in the LLM serving framework. The implementation is publicly available, contributing to the ongoing evolution of AI/ML applications in multi-turn contexts.