Show HN: Latent-free ternary LLM training (github.com)

BitBop has introduced a groundbreaking approach to training ternary language models, where weights are limited to {-1, 0, +1}, effectively eliminating the need for full-precision latent weights. This is achieved by utilizing a small bf16 flip momentum per weight, which significantly reduces the optimizer's memory footprint by approximately five times. As a result, larger models can now be trained on GPU setups with limited memory, such as a single RTX 3060. The significant shift to int2-packed weights also means that inference storage requirements are drastically decreased, offering a promising solution for memory-constrained deployment scenarios. The implications of BitBop's innovations are substantial for the AI/ML community, particularly in addressing the challenges of memory utilization in neural network training. The model has shown to outperform traditional training methods, specifically surpassing the Straight-Through Estimator (STE) in controlled comparisons, while achieving competitive results against float counterparts without extensive tuning. For instance, a 125M-class ternary model was able to achieve similar performance to a float GPT-2 model in specific benchmarks, reinforcing the viability of ternary models at lower parameter counts. This work not only paves the way for more efficient training techniques but also opens the door to new possibilities in developing lightweight AI models suitable for various applications.