Show HN: Why Rotating Vectors Makes Compression Beautiful (demos.connectai.blog)

A new technique known as PolarQuant, showcased on HN, optimizes the quantization of KV-cache vectors used in modern language models by addressing the prevalent outlier problem. Traditional quantization struggles with these outliers as their significant magnitude skews the representation of other dimensions, leading to poor reconstruction and inefficient storage. The innovative solution involves applying a random rotation matrix before quantization, which spreads the outlier energy uniformly across all dimensions. This results in a consistent data distribution, allowing the quantization process to assign slots more effectively and achieve minimal reconstruction error. PolarQuant enhances quantization efficiency by employing the Walsh-Hadamard Transform (WHT), which performs the necessary rotations using only additions and subtractions, achieving O(d log d) complexity, significantly faster than the O(d²) required by dense rotation methods. By incorporating random sign flips before and after the WHT, the technique maintains the randomness needed to ensure distinct outputs for similar input vectors. This streamlined approach not only preserves vital information during quantization but also maximizes the efficiency of space utilization in large language models, marking a substantial advancement in the field of AI/ML compression techniques.