AutoQuant KV: Automatic Mixed-Precision KV Cache Quantization (nvidia.github.io)

NVIDIA's Model Optimizer Team announced AutoQuant KV, a groundbreaking technology that automates mixed-precision quantization for key-value (KV) caches in AI models. This initiative allows for the flexibility of layer-specific precision, where the quantization process intelligently evaluates layer sensitivity to determine the most suitable bit format while adhering to a storage budget. By applying this technique to attention mechanisms, KV AutoQuant significantly compresses the memory footprint of models like Qwen3-8B, achieving a 64% reduction in modeled cache payload compared to traditional BF16 formats, with minimal impact on accuracy—just a 1.04% drop in RULER-128K performance. This advancement is particularly significant for the AI/ML community as it optimizes resource usage and enhances model efficiency without sacrificing performance. The KV AutoQuant process involves a systematic search over various K/V format combinations, taking into account different sensitivity levels and storage constraints, ultimately reducing costs associated with deploying large language models. The integration of KV AutoQuant with existing GEMM optimizations allows for a more comprehensive approach to model quantization, laying the groundwork for future innovations in AI deployment strategies and efficiency.