Tflops Gap: Why FP4 Moe Kernel Engineering Matters on Blackwell (huggingface.co)

NVIDIA's Blackwell architecture has introduced significant advancements in machine learning, particularly with its native support for FP4 quantization, promising impressive gains in memory bandwidth and throughput for large language model (LLM) inference. A recent benchmark comparing three Mixture of Experts (MoE) backend frameworks—SGLang, FlashInfer CuteDSL, and vLLM—demonstrated that SGLang achieved the highest peak throughput with 1262 TFLOPS using a GPT-OSS-20B model, outperforming vLLM by a notable margin of 145 TFLOPS. This performance gap highlights the importance of kernel engineering, as optimized kernels can drastically enhance inference times, especially at smaller batch sizes that are critical for interactive applications. Key optimizations in SGLang include kernel fusion, which reduces the number of memory passes and synchronization points, and Blackwell-specific CUTLASS schedules that leverage hardware capabilities for FP4 processing. Additionally, adaptive grid sizing ensures that the architecture remains efficient at low batch sizes, which are common in interactive tasks like chatbots. These advancements underscore not only the significance of hardware capabilities but also the impact of tailored software optimizations on achieving higher performance in AI applications, enhancing user experience through faster response times in real-time inference scenarios.