KVGuard: A KV Cache Safety Checker for LLM Serving Systems (arxiv.org)

A new tool named KVGuard has been introduced to enhance memory safety in large language model (LLM) serving systems by addressing critical vulnerabilities within their key-value (KV) cache management. Traditional serving systems using PagedAttention face unaddressed memory safety issues, such as double-free errors and use-after-free access. These issues often corrupt outputs and disrupt request isolation without crashing the system, rendering existing memory sanitizers ineffective. KVGuard addresses this gap by implementing a systematic memory-safety abstraction that tracks permission and initialization status for each KV block, enabling early detection of violations during the development phase rather than later stages when output corruption occurs. The significance of KVGuard lies in its ability to identify both historical and previously unknown safety violations while maintaining low overhead—a 3.0% to 7.4% increase in TPOT (Time to Process Output Tokens) and only 0.11% additional memory usage, significantly less than traditional solutions like NVIDIA Compute Sanitizer. By effectively mitigating risks associated with memory bugs and providing robust checks throughout the caching process, KVGuard represents a major advancement in the reliability and security of LLM serving systems, crucial for developers aiming to build safer and more robust AI applications.