Optimizing 124B MoE inference on a single Nvidia DGX Spark (developer.nvidia.com)

In a significant advancement for AI inference, Inclusion AI has optimized the performance of its Ling-3.0-flash model on an NVIDIA DGX Spark system, achieving a notable increase in decoding speed. By profiling the model, the team identified that the performance bottlenecks were mainly due to memory-bound operations associated with the mixture-of-experts (MoE) computation and the output-layer projection. They implemented three main techniques: Humming MXFP4 MoE kernels to reduce compute time, online FP8 quantization of the output layer to cut down weight-read volume, and multi-token prediction (MTP) speculative decoding to generate multiple tokens in fewer steps, collectively enhancing decoding throughput by up to 185.1%. These optimizations are critical as they improve the model's efficiency without compromising accuracy, which remained consistent at about 96.8% in accuracy assessments. The model, featuring 124 billion parameters and 42 layers, can now support complex tasks with a native context window of 256K tokens, expandable to 1M. The integration of these optimizations could lead to broader applicability in real-time AI applications, as they promise reduced latency and improved processing speed essential for interactive and resource-intensive scenarios. This underscores the potential for advanced architectures to handle increasingly demanding AI workloads while maximizing resource utilization.