Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models (arxiv.org)

Researchers introduced FS-DFM (Few-Step Discrete Flow-Matching), a new diffusion-style language model that makes long-text generation both fast and high-quality. Traditional autoregressive models are inherently serial (one token per pass) and diffusion language models, while parallel across positions, have required hundreds-to-thousands of iterative steps to match ARM quality. FS-DFM treats the number of sampling steps as an explicit parameter and trains the model to be consistent across step budgets so a few large updates land where many small updates would, enabling stable, few-step sampling without quality loss. Technically, FS-DFM combines a discrete flow-matching objective with a carefully designed update rule that moves probability mass in the correct direction without overshooting, plus strong teacher guidance distilled from long-run trajectories to shape short-run behavior. Empirically, an FS-DFM using only eight sampling steps matches the perplexity of a 1,024-step discrete-flow baseline when generating 1,024-token sequences with a similarly sized model—achieving up to 128× faster sampling and large latency/throughput gains. This work makes diffusion-based LMs practical for long-form generation, lowering inference cost and enabling real-time or high-throughput deployment while preserving controllability and stability.