Transformer Golf – The Unrolled Transformer (github.com)

A new experimental project called Transformer Golf has emerged, which aims to simplify the Transformer architecture to its core components necessary for in-context learning. Similar to "code golf," which solves programming challenges with minimal code, this initiative reduces architectural complexity by stripping away standard components like MLPs, LayerNorms, and biases. The project utilizes symbolic compilation through tools like torch.fx and egglog to visualize and verify simplified mathematical expressions, successfully demonstrating that a minimal Attention-Only network can achieve 100% accuracy on dynamic bigram prediction tasks. This development holds considerable significance for the AI/ML community as it highlights that a leaner architecture can perform effectively without excessive complexity. By proving that foundational components can operate independently to fulfill the demands of tasks typically assigned to more elaborate models, Transformer Golf introduces new paradigms for model efficiency and optimization in natural language processing. Notably, the unrolled architecture allows the GPU to perform efficiently with reduced memory footprint and computational demands, paving the way for future optimizations and innovations that prioritize core functions without unnecessary bloat.