Pretraining Large Language Models with NVFP4 (arxiv.org)

Researchers introduce NVFP4, a practical 4-bit floating-point scheme for pretraining large language models (LLMs) and demonstrate stable, long-horizon training at scale. Using a set of algorithmic safeguards, they trained a 12-billion-parameter model on 10 trillion tokens—the longest public 4-bit training run reported—and matched FP8 baselines in training loss and downstream task accuracy. This work addresses the pressing need to reduce the immense compute, memory, and energy costs of frontier LLM training by showing that sub-8-bit floating formats can be viable for large-scale pretraining. Key technical contributions include bounding block-level outliers with Random Hadamard Transforms (RHT), a two-dimensional quantization scheme that keeps forward and backward representations consistent, stochastic rounding to produce unbiased gradient estimates, and selective retention of some layers in higher precision for stability. Together these techniques mitigate the stability and convergence risks typically associated with aggressive quantization, especially over long token horizons. The results suggest NVFP4 can materially cut resource requirements and enable larger or more economical training runs, though practical deployment will depend on hardware support and continued engineering to identify which layers must remain high precision.