Show HN: Glq LLM quantization using E8 lattice (github.com)

A new Post-training weight quantization method for large language models (LLMs) called Glq has been introduced, leveraging E8 lattice codebooks. This approach encodes each group of eight weights as a 16-bit index into a 65,536-entry codebook, significantly enhancing quantization efficiency. By employing a Randomized Hadamard Transform (RHT) to decorrelate the Hessian, the method optimizes nearest-neighbor searches to maintain quality comparable to advanced techniques like QuIP# while surpassing GPTQ's performance. Notably, it enables weights to be stored at 2–8 bits per weight (bpw) without requiring the large dense weight matrix, using a custom fused CUDA kernel that performs matrix multiplication against the compressed indices directly. This development is significant for the AI/ML community as it enhances both the speed and efficiency of LLMs on GPUs while reducing memory bandwidth consumption. The integration process with popular frameworks like PyTorch and Hugging Face’s Transformers is straightforward via pip installation, allowing developers to easily implement Glq in their models. Early benchmarking shows that GLQ operates near bf16 throughput levels and manages to retain or surpass performance metrics across various tasks, making it a promising advancement for deploying high-performance, resource-efficient AI applications.