Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery [pdf] (research.nvidia.com)

A recent technical report introduces Quantization-Aware Distillation (QAD) as a novel approach to recovering the inference accuracy of NVFP4-quantized large language models (LLMs) and vision-language models (VLMs). By distilling knowledge from a full-precision teacher model to a quantized student model using KL divergence loss, QAD addresses limitations faced by traditional Quantization-Aware Training (QAT). The significance of QAD lies in its remarkable effectiveness and stability, particularly for models trained through complex multi-stage post-training processes, such as supervised fine-tuning and reinforcement learning. Additionally, QAD demonstrates resilience to data quality and coverage issues, achieving accuracy recovery even in scenarios with incomplete datasets. Key technical advancements outlined include the NVFP4 format's benefits, which provides superior arithmetic performance and memory efficiency compared to 8-bit formats. The report showcases QAD's superior alignment between quantized and high-precision models, confirming its effectiveness across multiple post-trained models like AceReason Nemotron and Llama Nemotron. This work positions QAD as a vital tool for practitioners seeking to optimize model performance while navigating the practical challenges of data availability and training complexity, thereby paving the way for broader adoption of efficient quantization in AI/ML implementations.