Compressing LLMs Without Compromise (www.sigarch.org)

Recent advancements in lossless compression for large language models (LLMs) promise to optimize inference by significantly reducing data movement without sacrificing accuracy. Traditional quantization techniques have limitations, leading to potential accuracy degradation and inconsistencies across various model architectures. In contrast, lossless compression approaches maintain model integrity while achieving an average data reduction of 1.4x, enabling models to fit within hardware constraints and improving inference speed. Key innovations include techniques like DFloat11, which compresses BF16 weight tensor exponents, and ZipServ, which streamlines data flow within GPUs for enhanced matrix multiplication processes. Invariant Bit Packing (IBP) adds further versatility by facilitating runtime decompression and allowing for the inclusion of generated KV cache in the compression process. These methods have shown promising results, with reductions in model sizes by up to 30% and improvements in inference performance, making lossless compression a critical area of research as the AI/ML community seeks to enhance multi-GPU deployments and address the significant overheads associated with data transfers. The exploration of hardware solutions, such as NVIDIA's new decompression engine, highlights the ongoing interest in integrating compression techniques into the entire data movement spectrum for LLM applications.