Show HN: A 155K-param transformer builds a map of a world it's never shown (ankur-chr.github.io)

A new project showcases a 155,000-parameter transformer model that autonomously constructed a mental map of its environment solely from a sequence of movement symbols, without ever being exposed to any coordinates or visual representations. By predicting the next move based on these symbols, the model developed a rich internal representation of its surroundings, which could then be accessed and manipulated through a "mind-reader" technique known as linear probing. Remarkably, when users altered the model's internal sense of location, it modified its behavior to reflect these false beliefs, demonstrating a causal rather than decorative understanding of its space. This development is significant for the AI/ML community as it points to the capacity of language models to create complex internal representations based on minimal input. The results show high accuracy in various tasks: a 98.8% success rate in position decoding, and 100% accuracy in predicting legal moves and responding to belief edits. Moreover, similar internal mapping patterns have been observed in larger models like Othello-GPT and Llama-class, suggesting a broader implication that even small models can reveal intricate mechanisms of representation in AI systems. This promising research opens avenues to explore how real-world maps of space and time could be rendered within larger language models in the future.