Speculative Decoding in vLLM on AMD GPUs (vllm.ai)

Recent advancements in speculative decoding for large language models (LLMs) have been demonstrated using the vLLM framework on AMD GPUs. This approach introduces a draft-and-verify mechanism that allows multiple candidate tokens to be proposed simultaneously, rather than producing them one at a time as in traditional autoregressive decoding. By evaluating several draft tokens in one verification pass, the system can increase output-token throughput significantly. The study explored various speculative drafting techniques, such as native Multi-Token Prediction (MTP), EAGLE-3, DFlash, and DSpark, each differing in how they integrate with the target model and generate candidate tokens. The significance of this development lies in its potential to optimize LLM serving at scale, addressing latency issues that arise from the conventional token-by-token processing. Benchmarks on AMD Instinct MI300X and MI355X GPUs showed varying effects on throughput based on the drafting methods and model architecture used. This progress not only enhances the performance of LLMs but also sets the stage for more complex and efficient architectures in AI applications, ultimately making AI solutions more accessible and responsive in real-world scenarios.