Breaking the 1.58-bit Barrier for Ternary LLMs (arxiv.org)

Researchers have made a breakthrough in optimizing Ternary Large Language Models (LLMs) by introducing a new storage layout called BITCOS, which effectively reduces the storage cost of weights in these models to as low as 1.485 bits per weight. Traditionally, ternary models use five-trit packing, which averages around 1.625 bits per weight. This inefficiency stems from the fact that a significant portion of weights (up to 51.5%) in ternary models are zero, leading to the need for a more adaptive encoding strategy. The significance of this development for the AI/ML community lies in its ability to enhance the efficiency of LLMs while maintaining performance. BITCOS employs a dense presence bitmap coupled with a compact sign vector, making it more space-efficient than previous packing methods across most tested models. The optimized format not only allows for more compact storage, but it also improves computational efficiency, with potential throughput increases of up to 1.28 times on matrix-vector multiplication tasks. This improvement is particularly relevant for modern processors and GPUs, streamlining LLM inference and enabling more efficient deployment of AI systems across diverse hardware platforms.