Pretraining Language Models via Neural Cellular Automata (hanseungwook.github.io)

Researchers have introduced a novel approach to pretraining language models using neural cellular automata (NCA) to generate synthetic data, potentially alleviating the ongoing data hunger of large language models. Traditional natural language corpora are expected to dwindle in quality and quantity by 2028, presenting challenges such as embedded biases and entangled reasoning processes. The team found that NCA-generated data, devoid of linguistic content, trains models to infer hidden rules from context, facilitating improved reasoning and long-range dependencies without relying on semantic shortcuts. Key findings show that, under matched token budgets, models pre-trained on NCA data consistently outperform those trained on natural language and other synthetic sources across various domains, achieving faster convergence and lower perplexity. Surprisingly, even when scaling natural language data to ten times the size of the NCA dataset, the latter still demonstrated superior performance. The ability to customize NCA complexity based on specific tasks—more straightforward patterns for programming code and complex dynamics for mathematical problems—opens new avenues for targeted model training. This research suggests a transformative strategy for developing language models that prioritize reasoning without inheriting human biases, shaping the future of AI with more effective and controlled training methodologies.