Canonical-basis realignment for Transformer LLMs: every hidden axis becomes independently measurable and controllable (github.com)

A recent announcement introduces the Canonical Basis for Language Models (CBLL), a transformative concept that employs a lossless coordinate transformation to enhance the interpretability of Transformer large language models (LLMs). This technique allows every hidden axis in the model's space to be independently measured and manipulated, shedding light on the complex interactions within these models. Notably, the CBLL enables researchers to isolate specific features and phenomena that are typically obscured in standard coordinate systems, like the behavior of positive and negative poles or the oscillation of model responses across layers. The significance of this development lies in its potential to revolutionize interpretability in AI/ML, making it easier to understand and control how models generate outputs. The CBLL's implementation—demonstrated on architectures like Qwen and SmolLM2—proves effective without altering the original model's behavior. Key technical features include the identification of shared alignment patterns across model layers and the introduction of spectral indices that quantify structural properties of LLMs. This advancement not only aids in model diagnostics and optimization but also inspires future research into alignment and multi-architecture performance, ultimately pushing forward the boundaries of AI interpretability.