XPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding (supercomputing-system-ai-lab.github.io)

XPress has been introduced as a novel competitive approach to improve the accuracy and efficiency of speculative decoding in AI language models. By enhancing the block-diffusion drafter mechanism, XPress integrates a lightweight causal refinement process that allows the model to address the challenge of non-causality inherent in traditional and diffusion drafting methods. This innovation allows for a significant increase in acceptance length, enhancing average performance by around 30% and decoding throughput by approximately 1.3 times, thanks to its ability to propagate causal dependencies across draft tokens without relying on sequential processing. This development is particularly significant for the AI/ML community as it strikes a balance between maintaining the speed advantage of parallel token generation while enhancing the contextual accuracy of the generated text. The approach retains a streamlined structure, adding only 80 million parameters to the existing dFlash drafter and preserving low overhead during inference. XPress leverages the drafter's hidden states to conditionally refine predictions, ensuring better linguistic coherence, which ultimately minimizes the risk of early token rejection during the verification phase. This refinement potentially sets a new standard in the ongoing quest for faster and more accurate language generation models.