Bw24 – from scratch rust+CUDA inference, every kernel tuned for sm_120a (github.com)

Bw24 has announced a groundbreaking inference engine designed from scratch using Rust and CUDA, specifically optimized for the RTX 5090 laptop, featuring the Blackwell sm_120a architecture with 24 GB of memory. Unlike other frameworks, each kernel is meticulously written and calibrated to maximize hardware capabilities, outperforming previous benchmarks like llama.cpp by 2.3 times in MTP speculative decoding across all tested Qwen models. The framework not only enhances inference speed but maintains output exactness, ensuring that faster results do not alter model responses. This development is a significant leap forward for the AI/ML community, particularly for applications requiring high-speed, single-user inference on NVIDIA's latest GPU architecture. The bw24 engine's rigorous approach includes precise kernel tuning and extensive compatibility testing, making it an ideal solution for those serving a single model to individual users. Its use of speculative decoding, advanced caching mechanisms, and a comprehensive suite of tools for validation positions bw24 as a critical resource for developers seeking efficient deep learning performance while ensuring fidelity and reliability in AI outputs.