Can SAEs Capture Neural Geometry? (www.goodfire.ai)

Recent research led by Usha Bhalla and Thomas Fel examines how sparse autoencoders (SAEs) can capture complex curved geometries in neural representations, a crucial step for enhancing interpretability in neural networks. The study acknowledges that while straight lines used in SAEs simplify the understanding of these complex structures, they also offer only limited views—akin to blind men experiencing an elephant in different ways. By recognizing three distinct methods—shattering, compact capture, and dilution—through which SAEs can represent geometric shapes, the authors lay the foundation for a new unsupervised pipeline aimed at uncovering the intricate geometries embedded within neural representations. This advancement is significant for the AI/ML community as it proposes a novel way to interpret and control neural networks, moving beyond isolated interpretations of individual features. Leveraging clusters of SAE features enables researchers to reconstruct the overall geometric structure, thus gaining a more comprehensive understanding of how networks operate internally. The researchers are also developing new architectures specifically designed for unsupervised manifold discovery, promising to enhance how we perceive and manipulate the complexities of neural architectures. This holistic approach aims to ultimately contribute to a deeper, mechanistic understanding of AI systems and improve their robustness and interpretability.