I built a pure WGSL LLM engine to run Llama on my Snapdragon laptop GPU (github.com)

A developer has created a novel Llama inference engine using Rust and WGSL, allowing it to run locally on any GPU without relying on CUDA or extensive frameworks. The engine utilizes the wgpu library to execute compute shaders for the entire transformation forward pass, demonstrating its capability with models like TinyLlama-1.1B. Significant for the AI/ML community, this development addresses the compatibility gap for integrated GPUs like the Adreno X1-85 in Snapdragon laptops, which traditional AI tools do not support. The engine's portability across various platforms (Windows, macOS, Linux) enhances its accessibility for developers and researchers. Key technical features of the engine include a minimalist architecture where each layer of the model is represented as an individual WGSL shader, allowing for increased transparency and hackability. It supports both half-precision and quantized modes to optimize memory usage, while its design enables straightforward modifications and experiments with the underlying code. Moreover, the engine’s architecture ensures strict context maintenance during execution, which is pivotal for preventing errors when generating low-level code. Although the engine is currently in alpha quality, lacking production-level features, it sets a foundation for further exploration and development in light of the growing landscape of local AI inference solutions.