NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models (arxiv.org)

Researchers have introduced NanoQuant, a groundbreaking post-training quantization method for large language models (LLMs) that enables efficient compression to binary and sub-1-bit levels. Traditional quantization techniques struggle with effective binary compression due to high data and computation demands, but NanoQuant addresses these limitations by reformulating the process as a low-rank binary factorization problem. Utilizing an efficient alternating direction method of multipliers (ADMM), it initializes and fine-tunes parameters to produce low-rank binary matrices and scales, leading to remarkable results in both memory usage and model performance. The significance of NanoQuant lies in its ability to achieve state-of-the-art accuracy while compressing models to drastically lower sizes, making it feasible to deploy large-scale LLMs on consumer hardware. For instance, it compresses the Llama2-70B model by an impressive 25.8 times in just 13 hours on a single NVIDIA H100 GPU, allowing such a complex model to run on standard 8 GB consumer GPUs. This advancement not only enhances accessibility for developers and businesses but also opens up new possibilities for the practical use of large models in various applications.