Performant C/CUDA inference engine for Qwen 3.6 35B on RTX 5090 / Blackwell (github.com)

A new C/CUDA inference engine, optimized for the Qwen 3.6 35B model, has been released, specifically designed for use with the NVIDIA RTX 5090 GPU. This engine is significant for the AI/ML community as it boasts superior performance metrics, including an extreme prefill throughput of 13.4k tokens per second and an impressive decoding speed of over 270 tokens per second. This hyper-optimized solution aims to outperform generic runtimes like llama.cpp by focusing on a dedicated model and GPU pairing, achieving faster results across various performance benchmarks. The engine incorporates advanced features such as hybrid architecture support for both attention and recurrent layers, dual-tier state management for efficient memory use, and a zero-dependency HTTP server compatible with OpenAI. Notably, it also employs intelligent KV caching for context restoration, significantly reducing state recovery time to just 3 milliseconds. With configurations for quantization that lower VRAM usage and maintain performance—while only slightly impacting accuracy—this development could set new standards for high-throughput, low-latency AI inference on modern hardware, making it a game-changer for developers and researchers working with large language models.