Ninfer: High-performance single-GPU inference (github.com)

NInfer has launched a specialized C++/CUDA inference engine optimized for two specific Qwen3.6 model checkpoints, allowing maximum performance on a single NVIDIA GeForce RTX 5090. This engine supports text, image, and video prompts, operating via a command line interface or APIs compatible with OpenAI and Anthropic services. Notably, NInfer offers impressive inference speeds, achieving up to 15,544 tokens per second for prefilling and 271 tokens per second for decoding on the larger 35B model with an extensive context capacity of 262,144 tokens. The significance of NInfer lies in its ability to deliver high-performance inference for advanced machine learning models without requiring a multi-GPU setup, thus making cutting-edge AI more accessible. By focusing on a closed set of model artifacts, NInfer enhances efficiency and simplifies usage scenarios for developers. The tool supports a wide range of features, such as speculative decoding and multimodal input processing, but is specialized for single-GPU execution, reinforcing its utility in edge deployments where computational resources are limited. For the AI/ML community, this development emphasizes the growing need for high-performance, resource-efficient solutions in model deployment and inference.