Native Rust/C++ LLM inference engine (github.com)

A new AI inference engine, "Paddock," has been launched, developed in Rust specifically for NVIDIA GPUs, featuring a comprehensive API compatible with OpenAI and Anthropic models. Unlike existing wrappers, Paddock is a native engine, integrating a custom scheduler, paged KV cache, memory management, and CUDA kernels directly within its codebase. The platform is designed to facilitate seamless deployment of open models in production environments, particularly on high-performance GPUs like the RTX 5090 and PRO 6000. Users can run two main binaries—paddock-runner for model execution and paddock for model management—ensuring simplicity in setup with only the NVIDIA driver required. Paddock aims to outperform existing engines like llama.cpp and vLLM in inference speed for models fitting within a single GPU. The engine currently supports various quantization formats, including native FP8 and several k-quant families, and can load weights from both GGUF and safetensors. Additionally, Paddock includes a built-in web studio for managing, comparing, and running models. With ambitious plans for tensor parallelism and continuous development, Paddock is poised to significantly impact the AI/ML community by enhancing model execution efficiency and accessibility for organizations.