Balanced Ternary Transformers: 93.8% Energy Reduction Using 1965 Soviet Research (zenodo.org)

A groundbreaking quantization method for transformer-based language models has been announced, leveraging concepts from 1960s Soviet research on balanced ternary systems. This approach constrains model weights to balanced ternary values {-1, 0, +1}, effectively eliminating the need for conventional floating-point matrix multiplication. Instead of utilizing multiply-accumulate operations, the method employs addition, subtraction, and skipping, resulting in a remarkable 93.8% reduction in energy consumption per inference. Furthermore, this innovation leads to a staggering 16x memory compression, bringing the storage requirement for a 7 billion parameter model down from 28GB to just 1.75GB. The significance of this development for the AI/ML community is profound. It not only enhances energy efficiency and memory usage but also promises a 48x improvement in theoretical throughput while maintaining 87-92% signal preservation. Additionally, the design introduces a mechanism for epistemic uncertainty, allowing models to abstain from making predictions on uncertain inputs, thus reducing the chances of AI hallucinations. Notably, this method is compatible with standard CPUs, eliminating the need for specialized hardware, and its full implementation is open-sourced, making it accessible for further research and applications in the field.