A Geometric Calculator Inside a Neural Network (www.goodfire.ai)

Researchers have identified a novel geometric addition module within the Llama 3.1 8B language model, allowing it to perform arithmetic on circular representations of numbers seamlessly. This module computes sums in a single forward pass, utilizing a unique approach that combines different modular circles to represent numbers. For example, to answer a query such as "What is 7 + 9?", Llama converts numbers into their circular forms, processes the calculations for each circle in parallel, and outputs the correct result through a geometric manipulation of these activations. This discovery is significant for the AI and ML community as it opens up new avenues for understanding how neural networks utilize geometric representations in their internal computations. By revealing that neural networks can effectively perform arithmetic on cyclical structures rather than linear ones, the study encourages a deeper exploration of neural geometry. This could lead to improved techniques for debugging, controlling, and designing AI models, enhancing their cognitive capabilities and generalization across diverse tasks. Such insights contribute to the overarching goal of creating more transparent and robust AI systems.