Discrete Diffusion in Large Language and Multimodal Models: A Survey (arxiv.org)

This survey synthesizes recent advances in discrete diffusion models applied to language and multimodal tasks, framing Discrete Diffusion Language Models (dLLMs) and Discrete Diffusion Multimodal Language Models (dMLLMs) as a viable alternative to autoregressive (AR) generation. Unlike AR models that generate one token at a time, discrete diffusion models use multi-token, parallel decoding with full attention and a denoising-based generation process. That design yields practical benefits—parallel generation, finer-grained controllability over outputs, and dynamic perception—while matching AR performance in many benchmarks and in some cases delivering up to 10× faster inference. Both industrial-scale proprietary systems and a growing open-source ecosystem demonstrate these gains, challenging the dominance of AR architectures for generative AI. Technically, the survey formalizes the discrete diffusion framework (denoising trajectories in token space), categorizes modeling choices, and reviews key methods for training, sampling/inference, and model compression/quantization. It also canvasses trust and safety concerns and emerging applications spanning text, vision–language, and biological sequence modeling. By compiling representative papers, code and demos, the authors highlight open challenges—scaling dynamics, sampling efficiency, alignment and evaluation metrics—and sketch research directions for deploying d(M)LLMs in production.