AXS-6: Adaptive EXponent Sharing – 6-Bit Training Format (github.com)

The AI/ML community has been introduced to AXS-6, a groundbreaking 6-bit numerical format that provides 4× the mantissa precision of FP8 while reducing memory usage by 21%. Developed with a custom Triton kernel, AXS-6 achieves impressive performance improvements, operating twice as fast as PyTorch's eager mode in certain tasks. This new format simplifies the training process by utilizing a shared exponent across blocks of values, eliminating the need for delayed scaling or loss scaling, which are often required in traditional FP8 implementations. As a result, AXS-6 ensures robust convergence, making it ideally suited for software-level quantization in neural networks. AXS-6’s innovative quantization strategy involves a 1024-entry lookup table that enables direct mapping of values, significantly streamlining the quantization pipeline. This method leads to 31% faster training compared to uniform quantization grids, making it particularly advantageous for memory-bound workloads. Benchmarks reveal that AXS-6 can reduce bandwidth requirements for gradient compression, presenting practical benefits for large-scale distributed training. While current evaluations suggest existing methods like BF16 still excel at smaller model sizes, AXS-6's efficiency becomes more compelling as model scales increase, highlighting its potential impact in advancing low-precision training techniques in AI.