Sky sphere representation in language models (29 Jul 2026) (arxiv.org)

In a groundbreaking study, researchers have discovered that certain open-source language models, particularly those around 100 billion parameters, possess a decodable representation of the night sky. This representation is significant as it is not merely an artifact, but rather a complex feature that emerges prominently when prompted with celestial queries. The analysis demonstrates performance with a $R^2$-score of 65-85% and a median angular error ranging from 12 to 21 degrees in leave-one-out testing, confirming that these models encode substantial variance pertaining to night sky maps. This finding is noteworthy for the AI/ML community as it highlights the capacity of large language models to capture intricate and high-dimensional geometric representations, marking a potential shift in how we understand model interpretability and feature manifolds. Specifically, the emergence of a curved high-dimensional irreducible feature manifold suggests new avenues for exploring the underlying mechanisms of language models and their ability to encode and retrieve complex spatial information beyond conventional flat representations. This advancement paves the way for future applications in astronomy and other fields requiring spatial reasoning from AI systems.