Differential Transformer V2 (huggingface.co)

Microsoft researchers have unveiled Differential Transformer V2 (DIFF V2), an enhanced architecture designed to optimize attention mechanisms in large language models (LLMs). Building on the original DIFF V1, this iteration effectively doubles the number of query heads while maintaining the same number of key-value heads. This innovative design allows for faster decoding speeds, crucial for memory-constrained LLM environments, as it reduces the need for custom attention kernels and minimizes the impact on throughput during pretraining. The significance of DIFF V2 lies in its ability to enhance training stability and reduce gradient spikes when using large learning rates, traits that were problematic in previous models. Notably, DIFF V2 eliminates the need for per-head RMS normalization, resolving issues with gradient magnitude and stability. The architecture is further enhanced by introducing a projected element that helps control context RMS, crucial for preventing attention sinks and maintaining training efficiency. Early experiments indicate that DIFF V2 achieves lower language modeling loss compared to standard Transformers, suggesting promise for future applications in long-context benchmarks and greater overall model efficiency.