KVarN: Native vLLM backend for KV-cache quantization by Huawei (github.com)

Huawei has announced KVarN, a new KV-cache quantization backend designed to enhance the efficiency of Long-Context Workloads using vLLM. KVarN significantly improves KV-cache capacity by 3-5 times while achieving up to 1.3 times the throughput of FP16 precision, maintaining comparable accuracy. This advancement allows systems to manage longer contexts and serve more simultaneous requests without compromising performance. Notably, KVarN is calibration-free and can be seamlessly integrated into vLLM with minimal setup, requiring just one flag to activate. The significance of KVarN lies in its ability to overcome the traditional trade-offs associated with KV-cache quantization methods, which often lead to reduced throughput and compromised accuracy. By employing a novel four-stage process involving rotation, normalization, and low-bit quantization, KVarN maximizes throughput while ensuring high accuracy on demanding production tasks. With the capability to function in float16 and an optimized architecture, KVarN positions itself as a crucial tool for developers in the AI/ML community, particularly those focused on enhancing model efficiency for tasks requiring extensive context management. This development marks a major step forward in practical applications of KV-cache quantization, making it more viable for real-world scenarios.