LRU is harder to beat than the KV-cache papers suggest (github.com)

A recent exploration into the effectiveness of Least Recently Used (LRU) cache policies has revealed that LRU may be more resilient than some recent literature suggests, particularly in the context of agentic AI workloads. An experiment replayed over 68,000 requests from real user sessions indicated that, contrary to the hypothesis that idle sessions could skew caching efficiency, the primary cause of cache misses under capacity pressure stemmed from tight loops of tool calls occurring mere seconds apart. It was found that the anticipated benefits of adjusting cache policies based on session activity, such as implementing alternative strategies to LRU, actually did not outperform LRU in real-world scenarios, prompting a reevaluation of cache management strategies for both performance and resource optimization. The significance of this finding lies in its implications for the design of caching mechanisms in AI systems, particularly those involving large language models (LLMs). As agentic systems increasingly demand higher efficiency, the data indicates that simply changing the evicting policy may not suffice if fundamental issues like cache capacity and session management remain unresolved. The study argues for a shift in focus towards developing capacity-aware caching strategies rather than solely relying on dynamic policies or idle-session predictions, potentially leading to advancements in cache architecture that can better accommodate the unique behaviors of modern AI applications.