LoPA: Scaling Diffusion LLM Single-Sample Throughput to 1000 TPS (zhijie-group.github.io)

The Lookahead Parallel Decoding (LoPA) algorithm has been introduced to enhance the inference speed of diffusion large language models (dLLMs), allowing for an unprecedented single-sample throughput of up to 1073.9 tokens per second. This advancement is particularly significant for the AI/ML community, as it overcomes the previous constraints of typical decoding strategies that only achieved 1-3 tokens per forward pass. By leveraging higher degrees of parallelism without sacrificing predictive performance, LoPA stands to revolutionize applications that rely on real-time or high-volume text generation. LoPA works by exploring multiple token filling orders (TFOs) during decoding, enabling it to create branches that assess potential outcomes. This strategy is based on the observation that the degree of parallelism fluctuates with the prediction confidence during inference. As a result, LoPA employs a confidence-based metric to evaluate and select the optimal branches for sampling, ensuring that it fills more tokens effectively with each pass. Integrated with the D2F model, LoPA increases throughput while maintaining or improving generation quality, highlighting its potential for broader adoption in various AI applications, such as coding and mathematical tasks. Future developments may include the adaptation of LoPA to other models and the introduction of a new framework named Diffulex, which aims to further streamline and enhance dLLM inference.