Can I Buy Your KV Cache? (arxiv.org)

A recent proposal highlights a significant efficiency breakthrough in the AI/ML community, focusing on the computation of key-value (KV) caches during AI agent operations. Currently, many AI agents redundantly recompute the KV cache for the same documents from scratch, resulting in massive computational waste. The proposed solution suggests that publishers precompute and sell access to these KV caches, allowing subsequent agents to load them directly and bypass the compute-intensive "prefill" process. This approach has shown to be 9-50 times more efficient in terms of resource use, particularly for longer documents, ultimately delivering substantial cost savings. The implications of this proposal are immense—serving a single popular document could cost up to $1.5 million in recomputation, while reuse would only require about $30,000. Furthermore, hosting the KV caches on provider servers circumvents expensive data transmission costs associated with transferring caches. The envisioned "agent-native prefill CDN" could revolutionize the accessibility and efficiency of AI workflows, presenting an opportunity for dramatic cost reductions while maintaining precision. As the paper concludes, addressing challenges like KV compression and establishing a payment framework could unlock millions in potential savings for AI document processing.