Tiny inference engine in pure Rust, CPU-only, with TUI visualization (github.com)

A new inference engine for the MiniCPM5-1B model, designed in pure Rust, has been released, offering a CPU-only solution for understanding large language models. Developed as a minimal, single-file implementation, it avoids the complexities of GPU reliance and KV-cache, instead focusing solely on the core mechanics of the model. This project, inspired by tiny-vllm, aims to facilitate learning by directly translating procedural knowledge into the code structure, featuring around 1,800 lines of organized yet dense code. Significantly, this engine includes a terminal user interface (TUI) that visualizes real-time model interactions, making it easier to grasp how the model operates. It employs various advanced techniques including rotary position embeddings, multi-head attention with Grouped Query Attention, and an MLP with SwiGLU activation. Users can download the MiniCPM5-1B model from HuggingFace and visualize tensor operations including attention maps and logits during inference. This hands-on approach not only enhances educational efforts in AI and ML but also delivers a robust tool for practitioners interested in low-level model mechanics without the need for specialized hardware.