Over 1k tok/s on an RTX 5090 with Qwen3 0.6B (blog.alpindale.net)

This weekend, an impressive breakthrough was achieved in optimizing AI model inference speed with the Qwen3-0.6B model on the RTX 5090 GPU, where a single CUDA megakernel managed to process up to 1,000 tokens per second in bfloat16 precision. This performance marks a significant increase compared to previous optimizations that peaked at 530 tokens per second on an RTX 3090. The achievement highlights the importance of fine-tuning kernel architecture, given that the new RTX 5090 hardware offers over twice the streaming multiprocessors and increased memory bandwidth compared to its predecessor. The key to this enhanced performance lies in several technical innovations within the kernel’s architecture. These include reducing synchronization overhead through the implementation of atomic barriers, optimizing redundant computations in layer normalization, and utilizing effective memory access patterns through prefetching strategies. Such optimizations not only minimize latency but also ensure efficient memory bandwidth utilization, which is critical for deep learning inference tasks. This breakthrough not only demonstrates the potential of the RTX 5090 GPU for high-speed AI applications but also provides valuable insights and source code for the AI/ML community to replicate and build upon, pushing the boundaries of real-time model performance.