Show HN: UltraCompress – first mathematically lossless 5-bit LLM compression (github.com)

UltraCompress has announced a groundbreaking compression tool capable of achieving mathematically lossless compression for large language models (LLMs), specifically targeting HuggingFace transformer checkpoints. By reducing model sizes from 1.7 billion to 405 billion parameters while maintaining a perplexity degradation of less than 1.5%, UltraCompress enables users to run large models on a single 32GB consumer GPU. This significant advancement allows even models that exceed GPU memory limits to be processed, facilitating broader accessibility and deployment of powerful AI models. The technology employs a two-phase process: initially streaming decoder layers while caching hidden states, and then applying a low-rank correction based on a per-layer V18-C training methodology. Several models, including Qwen3, Mistral, and Llama, have been tested, showing impressive performance with a mean perplexity ratio below 1.013. This means that the compressed models retain high quality despite their reduced size. With support for any CUDA GPU with a minimum of 16GB VRAM and a straightforward installation process, UltraCompress represents a significant leap forward in making sophisticated AI accessible to a wider audience, pushing the boundaries of efficient AI deployment.