I built a language model out of cellular automata (ssenthilnathan3.github.io)

A recent experiment exploring the capabilities of a language model built from cellular automata has revealed intriguing insights into local conversation rules in machine learning. The model, resembling a neural cellular automaton, utilized a grid of cells that communicated with their neighbors through learned vectors, bypassing traditional methods like transformers or attention mechanisms. Initial results showed that the model could generate recognizable phrases after training, but it struggled with long-distance information transfer, leading to repeated outputs without meaningful variation. Key findings indicated that while the model could transport bits over long distances in simpler tasks, it failed to do so effectively in more complex language modeling scenarios, where the predictive context complicated the data flow. This research is significant for the AI/ML community as it raises questions about the limits of localized information processing in neural networks and challenges assumptions about model architecture and memory. It suggests that even when a model can physically transfer information across a grid, it may resort to shortcuts in tasks where the immediate context offers predictability, obscuring its true potential for knowledge retention and versatility. By highlighting the nuanced interplay between model design, training objectives, and the nature of tasks, this work provides valuable insights for future studies aiming to enhance language understanding and memory in machine learning systems.