When Accuracy Meets Parallelism in Diffusion Language Models (hao-ai-lab.github.io)

Recent developments in Diffusion Large Language Models (dLLMs) highlight their potential advantages over autoregressive (AR) models, including parallel decoding and random-order generation. However, a study reveals a significant accuracy-parallelism trade-off in current dLLMs: while they can process multiple tokens per forward pass, this often results in accuracy degradation. The new metric, Accuracy Under Parallelism (AUP), has been introduced to better assess this trade-off comprehensively. The findings suggest that despite impressive throughput metrics reported by closed-source models like Gemini Diffusion, open-source diffusion models still lag behind in accuracy compared to strong AR models, particularly those enhanced with speculative decoding techniques. The d3LLM framework aims to reconcile these issues by improving both accuracy and parallelism. It utilizes a pseudo-trajectory distillation process and a multi-block decoding method with a cache refresh to enhance performance. The study underscores that although dLLMs exhibit considerable speed advantages, they do not currently dominate the landscape when speed and accuracy are considered together. This research emphasizes the need for improved methodologies in the diffusion approach to elevate overall model performance while maintaining high accuracy—a critical goal for the AI/ML community moving forward.