The MTP sweet spot moves as context fills: full-context benchmarks on Strix Halo (thefrontierlab.ai)

A recent benchmark by kmarble highlighted significant performance differences between computing contexts using ROCm and Vulkan backends on the Qwen3 models. The study showed that while ROCm initially excels by processing at 46 tok/s, its performance plummets to 16.6 tok/s when the context is filled with 76k tokens—a stark 64% drop. In contrast, Vulkan only experiences a modest drop in speed, from 32.7 to 28.9 tok/s. Notably, enabling the Multi-Token Prediction (MTP) feature allows ROCm to regain some speed, pushing it to 37.5 tok/s, making it the recommended setup. This divergence highlights the importance of optimizing context management in AI applications, particularly as workloads evolve to utilize larger context windows. For the broader AI/ML community, this finding underscores a critical shift in understanding performance optimization. The way context depth influences the efficacy of draft mechanisms indicates that optimal draft depths must be recalibrated as contexts grow. For instance, in kmarble's tests, the ideal draft depth shifted from n=2 to n=1 as full context was filled, suggesting that risks associated with deeper drafts increase significantly at higher context lengths. Furthermore, the implications of prefill costs underline the need for efficient context handling, particularly for interactive applications where large datasets are processed. This research paves the way for more efficient model deployment strategies and enhances performance understanding across different hardware configurations.