UlaCache: Unified Logical Addressing for Sparse LLM Serving (arxiv.org)

UlaCache, a new KV cache management system, has been introduced to enhance the efficiency of large language model (LLM) inference by addressing the challenges associated with long contexts and sparse attention. By establishing a unified logical address space that bridges GPU and optional CPU memory, UlaCache allows for efficient allocation and retrieval of the key-value caches independently of their physical locations. This system implements a Parallel Bucketed LRU (PB-LRU) algorithm for managing GPU memory shared across different requests, optimizing both memory usage and access patterns during inference. The significance of UlaCache lies in its potential to substantially increase decoding throughput, achieving performance improvements of up to 2.51 times compared to existing state-of-the-art methods, depending on the configuration. With improvements in data-parallel and tensor-parallel attentions, UlaCache not only reduces memory costs but also minimizes memory traffic, making it a vital tool for future advancements in AI/ML applications that involve large-scale LLMs. This innovation fosters more effective resource utilization in AI systems, paving the way for more scalable and efficient language processing solutions.