A pure Rust LLM engine (~2k LOC) with live GPU/CPU brain migration (github.com)

A new pure Rust Transformer inference engine, comprising approximately 2,000 lines of code, has been developed to run the Llama-3.2 model with impressive speed, reaching around 40 tokens per second on an Nvidia Tesla T4 GPU. This innovative project features a groundbreaking live brain migration capability, allowing the engine to transition execution between GPU VRAM and CPU RAM in real time—specifically, mid-generation. This means that as it generates Python code, it can fluidly switch its processing between the GPU and CPU without losing any context, effectively enhancing performance and flexibility in AI applications. This development is significant for the AI/ML community as it showcases the potential of pure Rust programming to achieve high performance in machine learning tasks while providing cross-silicon portability. The same engine can operate seamlessly across various hardware platforms, including discrete Nvidia GPUs, integrated graphics from Intel, and multi-core CPUs, without any code modifications. Moreover, it leverages modern technologies such as Vulkan 1.3 for hardware dispatch and allows for efficient parallel execution by utilizing the Rayon library on the CPU. The minimalist implementation encapsulates essential components like a Safetensors parser and tokenization directly in Rust, representing a significant shift towards more efficient and versatile AI development frameworks.