BERT Is Just a Single Text Diffusion Step (nathan.rs)

Research shows that discrete text diffusion—used by experimental models like Google DeepMind’s Gemini Diffusion—is essentially a generalization of masked language modeling (MLM): BERT-style MLM is just a single denoising step at a fixed mask rate. The author demonstrates this with a proof-of-concept: fine-tuning RoBERTa-base (HuggingFace) as a diffusion model by sampling variable mask rates (mask_probs from 1.0 down to 0.1) across ~10 denoising steps, preserving a 16-token prefix for conditioning, and iteratively predicting and re-masking the remaining 240 tokens in a 256-token sequence. Training optimizes a sum of denoising losses over timesteps, turning the MLM objective into a full generative procedure—DiffusionBERT independently reached similar conclusions. Technically, the implementation uses a custom diffusion_collator to apply batch-level mask probabilities, standard top-k/top-p sampling for each masked position during each step, and runs on an H200 (30 minutes of fine-tuning) to produce surprisingly coherent passages. Compared to GPT-2 it’s a bit slower (~13s vs ~9s) and slightly less polished, but validates that encoder-only models can generate whole blocks of text without architectural changes. The result unifies diffusion and MLM viewpoints and suggests practical avenues (AR-Diffusion, skip-step strategies, optimized sampling) to improve speed and quality.