Linear Representations and Superposition (ternarysearch.blogspot.com)

Recent research highlights the importance of mechanistic interpretability in large language models (LLMs), focusing on two concepts: the linear representation hypothesis (LRH) and superposition. The LRH posits that certain concepts are represented linearly in LLMs, much like the well-known relationship in word embeddings where vector manipulations reveal semantic relationships. The paper by Park et al. formalizes this idea by introducing a model where input and output representations behave similarly under intervention, demonstrating that concepts can be manipulated linearly in both embedding and unembedding spaces. The empirical research using Llama 2 suggests that various concepts can be modeled effectively within this framework, revealing underlying structural relations among concepts. On the other hand, the phenomenon of superposition addresses the challenge of representing numerous language features within the limited dimensionality of LLMs. Research from Anthropic shows that while features can interfere in low-dimensional contexts, higher-dimensional spaces allow for almost-orthogonal arrangements through non-linear activations, like ReLU. This flexibility enables models to navigate the complexities of language representations efficiently. Aspects such as sparsity and the geometrical organization of embeddings highlight a deeper relationship between mathematical structures and the interpretability of intelligent systems, offering valuable insights that could enhance our understanding and application of LLMs.