Smallest transformer that can add two 10-digit numbers (github.com)

Researchers have successfully built the smallest transformer capable of accurately adding two 10-digit numbers, achieving over 99% accuracy on a set of 10,000 test cases. This initiative, known as "Addition Under Pressure," had participants leveraging creative strategies to minimize the number of parameters in their transformers while ensuring they could perform addition as an autoregressive process. The leading model, developed by a user named alexlitz, utilized just 36 parameters and showcased notable innovations, including ALiBi positional encoding and a sparsely embedded architecture, resulting in 100% accuracy. This achievement is significant for the AI/ML community as it demonstrates the potential for extreme architectural efficiency in neural networks, particularly transformers. It raises fundamental questions about the minimum complexity necessary for fundamental operations like addition. By exploring both trained and hand-coded weight setups, this work encourages innovative approaches to model design, prompting further research into parameter efficiency and applicability to other tasks. Key techniques such as rank-3 factorization and customized positional encodings have emerged, contributing to what might be a transformative understanding of deep learning architecture optimization.