petite-vllm Part 2: KV Cache & Paged Attention (kristenmcintosh.dev)

In part two of the petite-vllm series, significant enhancements have been made to the handling of Key-Value (KV) caching and attention mechanisms for large language models (LLMs). The introduction of KV caching allows models to avoid redundant computations during token generation by storing previously computed Key and Value projections. This optimization drastically reduces computational requirements, scaling linearly with sequence length, which translates to a substantial reduction in processing time. For instance, in the Qwen3-0.6B model, the required size for K/V projections reduces drastically, showcasing its effectiveness, leading to a more efficient autoregressive loop. Additionally, the concept of Paged Attention has been introduced, derived from operating system paging algorithms to address memory fragmentation within the KV cache. Instead of flat pre-allocations, this method segments the KV cache into fixed-size pages, efficiently utilizing memory by dynamically assigning blocks to sequences as needed, thus minimizing internal and external fragmentation. While this does not decrease total memory usage, it significantly enhances memory management efficiency, critical for systems handling numerous concurrent sequences. The implementation details demonstrate a robust structure incorporating data classes for block management, enhancing scalability and performance in LLM applications.