GSD-LQ: Zero-calibration sub-4-bit compression and 7B in 3.78 GB VRAM (github.com)

GSD-LQ has announced a transformative zero-calibration compression engine that enables sub-4-bit model compression, allowing for the execution of 7 billion parameter models in under 4 GB of VRAM. This model-agnostic tool can compress any open-weight transformer checkpoint, such as Llama and Mistral, using just a standard CPU, which significantly streamlines the deployment process for large AI models. The technology's lightweight design eliminates the need for complex GPU Hessian calibration, making advanced machine learning models more accessible for various applications. The implications for the AI/ML community are profound, as this innovation not only enhances memory efficiency but also sustains performance levels comparable to heavier models. The GSD-LQ engine operates on any OS and supports a wide range of GPUs, effectively lowering the barriers for researchers and developers working with resource-intensive AI applications. With impressive compression metrics—averaging around 3.8 bits/weight while maintaining high fidelity—the GSD-LQ tool sets a new benchmark for model efficiency, potentially revolutionizing how developers manage and deploy large transformer models in resource-constrained environments.