Two RTX 5090 hosts sharing a KV cache over Ethernet (viiwork.pulu.app)

A recent demonstration showcased the kv_share capability, enabling the sharing of key-value (KV) caches between two RTX 5090 GPU hosts over Ethernet. This innovation addresses a common bottleneck in large language model interactions, where losing the KV cache leads to time-consuming re-prefilling of entire prompts before generating new tokens. By allowing the cache to be shared across hosts, the next turn in a conversation can resume smoothly, even when a session must switch due to host load, drastically reducing response times. This advancement holds significant implications for the AI/ML community, particularly for real-time coding agents and conversational AI applications. The experiment revealed that using kv_share can keep the session "warm" — allowing for faster interactions — with rates of warm switches improving from 93.8% to 96.8% when using a 10 GbE connection compared to standard 1 GbE. The direct transfer method demonstrated enhanced efficiency, with replication times dropping significantly, underscoring the importance of network speed in optimizing the performance of distributed AI systems. As AI models grow in complexity, such enhancements in cache management and session continuity are pivotal for improving user experience and response times in AI-driven applications.