Continuously Augmented Discrete Diffusion Model (arxiv.org)

Researchers introduced Continuously Augmented Discrete Diffusion (CADD), a new framework for categorical generative modeling that removes the “information void” created when discrete diffusion models collapse unknown tokens into a single [MASK] state. Instead of mapping unobserved tokens to an absorbing token, CADD pairs the discrete state space with a continuous latent diffusion: masked tokens are represented as noisy, informative vectors that are gradually corrupted and then denoised alongside the discrete variables. During each reverse step the continuous latent acts as a semantic hint to guide discrete denoising, and the method is designed to slot into existing discrete diffusion training pipelines without architectural overhaul. Technically, CADD yields graded corrupted states and introduces a controllable knob at sampling: the strength and estimator used for the continuous latent determine a trade-off between mode-coverage (diversity) and mode-seeking (context fidelity). Empirically the paper reports consistent qualitative and quantitative gains over strong mask-based discrete diffusion baselines across text generation, image synthesis, and code modeling. The approach promises better semantic coherence and more controllable sampling behavior for categorical generative tasks, while remaining compatible with existing discrete-diffusion tooling—potentially improving performance where discrete masks previously discarded useful contextual information.