VindexLLM – Windows Vulkan GPU GGUF LLM inference engine written in Delphi 12 (github.com)

VindexLLM has emerged as a notable GPU-accelerated inference engine for large language models (LLMs) built entirely in Delphi, utilizing Vulkan compute shaders. This engine supports standard GGUF model files and executes the complete transformer forward pass on the GPU without requiring Python, CUDA, or external runtimes—only the Vulkan library is needed, which is included with modern GPU drivers. By performing heavy computations solely on the GPU, VindexLLM allows for seamless text generation and multi-turn conversations, supported by a SQLite-backed memory system for persistent context. This development is significant for the AI/ML community as it addresses common dependencies and limitations associated with existing LLM inference systems that often rely on NVIDIA's CUDA, requiring large toolkit installations and limiting compatibility. VindexLLM promotes an inclusive approach by functioning on any modern GPU, reducing installation complexity, and improving performance. Key technical details include the use of 34 layers of attention and feedforward networks executing on the GPU, efficient memory-mapped model loading, and TurboQuant, a novel 3-bit quantization approach ensuring high-quality output while minimizing memory requirements. Overall, VindexLLM presents a powerful, streamlined alternative for developers looking to implement efficient LLM applications without the typical barriers of existing frameworks.