Error Certificates for KV-Cache Eviction via Randomized Design (arxiv.org)

A recent study presents a novel approach to key-value (KV) cache eviction in machine learning models by introducing randomized designs that provide error certificates. Traditional deterministic eviction methods prioritize top tokens based on importance scores but fail to identify or manage the consequences of evicting certain values, which can lead to unpredictable increases in attention-output error. The new method employs Poisson-sampling to ensure identifiable and consistent estimations of cache-related errors, resulting in a powerful tool that separates cache-induced failures from inherent model issues. This development is significant for the AI/ML community as it enhances the reliability of model outputs during the serving phase by providing a more accurate understanding of how cache management impacts performance. Notably, the empirical validation showcases high-certainty coverage (0.97) without sacrificing accuracy, while revealing practical insights on resource allocation during runtime. The study also emphasizes that while randomized eviction improves attribution capabilities, it does not necessarily improve prediction accuracy, marking a crucial advancement in the ongoing effort to refine model serving strategies and improve performance metrics in real-world applications.