When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers (arxiv.org)

Recent research highlights challenges with using remote key-value (KV) caching in long-context language model (LLM) serving, particularly with the GLM-5.3 model deployed across 8xB300 nodes. While reusing the attention KV cache traditionally reduces prefill costs, enabling a shared remote tier significantly slowed response times, with the p99 latency skyrocketing from 3.96 seconds to 17.3 seconds. This study reveals that responding to requests from the remote cache took nearly three times longer than recomputing shorter prompts, emphasizing the limitations of the current setup where the bandwidth to the remote tier could not compensate for the inherent delays. To address these performance issues, the authors propose several innovations: sharding the KV store across processing ranks, partitioning the remote tier to improve bandwidth, and implementing asynchronous loading. These adjustments dramatically reduced p99 latency from 83.32 seconds to 0.67 seconds in hit-heavy scenarios, while prefill throughput increased by up to 34%. By submitting these enhancements to the LMCache upstream, the research presents a significant step towards efficient scaling of remote KV tiers, which could greatly benefit the AI/ML community by optimizing the performance of large-scale LLMs.