Is there even a ground-truth for LLMs' internal representations? (www.lesswrong.com)

Anthropic has introduced the "Jacobian Lens" (J-Lens) in their recent paper, which seeks to decode the internal representations of Large Language Models (LLMs) through a novel geometric approach. This builds on previous work with various lenses, like the Logit Lens and Tuned Lens, but challenges the conventional understanding of ground-truth meanings within LLMs. The J-Lens suggests that the meaning of an internal state should be derived intrinsically from the model's geometry rather than relying on external prompts or transformations, positioning the analysis within the framework of piecewise-linear functions typical of deep neural networks. This work is significant for the AI/ML community as it addresses a key epistemic challenge: understanding what hidden representations in LLMs truly signify. The J-Lens method focuses on identifying the label associated with the geometric region a vector occupies, aiming to produce a more principled interpretation of hidden states. By eliminating the inter-region transport through strategic adjustments in attention mechanisms, the J-Lens offers a fresh methodology for interpreting neural activations that could reshape our understanding of model behavior and representation learning, ultimately fostering more robust applications of LLMs in real-world scenarios.