Show HN: A mini paged-KV and prefix-cache scheduler (learning inference engine) (github.com)

A new project titled "mini paged-KV and prefix-cache scheduler" has been introduced as an experimental inference engine demonstrating a variety of innovative features for minimal LLM inference. Key components include a paged key-value (KV) cache with a block size of one, a radix/trie-based prefix cache that enhances shared prompt prefix efficiency, and a scheduler that manages KV capacity with admission control and batching. The project is built with practical benchmarking tools and emphasizes a straightforward implementation to aid learning in the area of large language model infrastructure. This initiative is significant for the AI/ML community as it showcases an accessible way to experiment with advanced caching and scheduling concepts, which are crucial for efficient model inference in real-time applications. By utilizing recent advancements like FlashAttention and a straightforward scheduling policy, it aims to make these technologies more understandable. The performance claims are notable, reaching 1990 tokens per second on an RTX 4070 with 80,000 allocated blocks, indicating strong potential for further exploration and optimization in LLM applications. This lean architecture could serve as a foundation for educators and developers seeking to deepen their understanding of LLM inference engineering.