HyperCache – Distributed KV-cache quantization and relative RoPE reuse (github.com)

HyperCache has been introduced as an innovative distributed key-value (KV) cache management system aimed at optimizing multi-node inference for large language models (LLMs). Traditional methods for KV caching, such as local prefix caching and disaggregated architectures, encounter significant bottlenecks when dealing with shared prompt tokens and inter-node data access. HyperCache addresses these issues by acting as an external cache layer that efficiently caches shared tensors and optimizes data transfer through advanced techniques like wire quantization and causal hash indexing. This development is crucial for the AI/ML community as it enhances the operational efficiency of LLMs, enabling faster and more scalable inference without the need for extensive infrastructure changes. HyperCache achieves significant reductions in latency—from over 500 ms in traditional methods down to as low as 22.4 ms for 16k tokens—by implementing per-head quantization and leveraging protocols like RDMA for rapid data streaming. Additionally, it utilizes relative RoPE (Rotary Position Embedding) shifting, allowing cached blocks to be reused effectively across different token offsets while maintaining accuracy, thus minimizing redundant computations in deep learning workloads.