Attention Sinks in Diffusion Language Models (twitter.com)

Researchers studying diffusion-based language models have identified a brittle failure mode they call "attention sinks": tokens or positions that gradually attract disproportionate attention across denoising steps, causing the model to lock onto a small set of tokens and collapse generation diversity. The paper analyzes attention dynamics in diffusion LMs (which iteratively denoise continuous token representations) and shows attention mass concentrates into a few sink nodes due to timestep conditioning, softmax saturation, and interaction between residual connections and attention heads. Experiments on synthetic and real-world text tasks reveal sinks degrade long-range dependencies, reduce sample diversity, and make guidance/control (e.g., retrieval or prefix conditioning) brittle. Technically, attention sinks arise from repeated re-weighting of queries/keys across diffusion timesteps and can be mitigated by architectural and training fixes: per-timestep layer normalization or learned timestep embeddings that decorrelate keys, softmax temperature or entropy-regularized attention to prevent head collapse, targeted attention dropout, and small changes to initialization and residual scaling. The finding matters because diffusion LMs are gaining traction for controllable and energy-based text generation; understanding and fixing attention sinks improves stability, fidelity to conditioning signals, and robustness of long-context reasoning, and suggests new diagnostics (attention entropy over timesteps) for future diffusion-language research.