Lossless compression of LLMs to 12-bits (github.com)

A new library named NF12 has been introduced, offering a lossless 12-bit encoding for BF16 weight matrices that significantly enhances the efficiency of large language model (LLM) inference. Developed by Together AI and detailed in the paper "NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight Compression," NF12 is optimized for NVIDIA GPUs from the Ampere architecture onward. This encoding method notably reduces memory consumption and supports faster processing speeds, with benchmarks showing performance improvements in token generation across various models, particularly when compared to traditional BF16 encodings. The NF12 library is significant for the AI/ML community as it allows for more efficient deployment of large models in environments with limited computational resources, making state-of-the-art LLMs more accessible. Technical specifications require a compatible NVIDIA driver, CUDA toolkit 13.0 or later, and PyTorch 2.10 or higher. The efficient memory architecture can handle complex operations like linear transformations while ensuring that the integrity of original model weights is maintained during encoding and decoding processes. This advancement paves the way for improved model scalability, reduced latency, and potentially transformative benefits in AI applications requiring high-performance LLM deployments.