Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing (arxiv.org)

A recent study introduces "Stochastic KV Routing," a novel approach aimed at enhancing the efficiency of serving transformer language models through innovative cache sharing strategies. By focusing on the depth dimension of Key-Value (KV) caches, the researchers address the significant memory overhead that comes with KV caching during autoregressive generation. This method departs from traditional compression techniques, showing that allowing layers to share cache resources can optimize memory usage without deteriorating model performance or throughput. The key innovation lies in a strategic training method where layers randomly alternate between utilizing their own KV states and those from previous layers. This stochastic process not only adapts the model to various depth-wise cache sharing frameworks but also demonstrates a potential regularization effect for larger models, leading to improved performance in data-constrained scenarios. The findings suggest that such depth-wise cache sharing can significantly lower memory requirements while maintaining or even enhancing the efficiency and effectiveness of transformer models, marking an important advancement in the AI/ML landscape as it addresses both computational costs and model robustness in real-world applications.