In search of wasted bits: how much information do LLM weights carry? (fergusfinn.com)

Recent research has uncovered significant "slack" in the storage of large language model (LLM) weights, particularly when using formats like bfloat16 (BF16). The study utilized Shannon entropy to analyze weight distributions from multiple open-weight models, revealing that the current allocation of bits does not fully utilize the available information content. Specifically, it found that BF16 weights average only about 10.6 bits of entropy per element despite having a 16-bit allocation, with around a third of that budget being wasteful, particularly in the exponent component. This indicates that while the mantissa and sign are effectively utilized, the exponent remains largely underused due to the concentrated distribution of weight magnitudes. This discovery is crucial for the AI/ML community as it highlights that there is still room for improvement in model efficiency and data storage optimization. As models continue to grow in size and complexity, understanding the distribution of weights can lead to better quantization methods that reduce memory usage while maintaining or enhancing computational efficiency. Exploring narrower formats, like FP8 and FP4, may further reduce slack, but they also necessitate shifts in how models generate weight distributions to adapt to tighter bit allocations. This could significantly impact the design of future models and their deployment in resource-constrained environments, ultimately improving the speed and efficiency of LLM inference.