Snow HN: ~950 line inference engine, on par with vLLM (github.com)

A new inference engine, known as Snow HN, has been introduced, comprising approximately 950 lines of code and built from scratch to work with OpenAI's gpt-oss-120b model on a single NVIDIA H100 GPU. This minimal and extensible engine aims to demonstrate that creating a performant architecture for large language models (LLMs) is feasible. Notably, it showcases advanced techniques like asynchronous batch processing, CUDA graph functionality, and a slot-based key-value cache system, all designed to maximize GPU efficiency and throughput. Significant for the AI/ML community, Snow HN provides a simplified yet powerful framework for researchers, students, and developers looking to experiment with LLM inference techniques. It serves as an accessible starting point to explore the intricacies of model inference, offering continuous batching and memory-efficient attention mechanisms such as Flash Attention 2. The codebase encourages modification and experimentation, making it an attractive alternative for research labs and learners who previously relied on more complex engines like vLLM. With its efficient design, Snow HN invites innovation and practical learning in the burgeoning field of AI inference engines.