Scaling Latent Reasoning via Looped Language Models (arxiv.org)

Researchers have unveiled Ouro, a series of pre-trained Looped Language Models (LoopLM) that enhance reasoning capabilities by integrating iterative computation during the pre-training phase. Unlike traditional Large Language Models (LLMs), which typically rely on explicit text generation for reasoning after training (like chain-of-thought techniques), Ouro incorporates reasoning and knowledge manipulation directly into its architecture. This approach allows the models to process 7.7 trillion tokens, resulting in performance that rivals larger state-of-the-art models (up to 12 billion parameters) across various benchmarks. The significance of Ouro lies in its advanced ability to manipulate knowledge more effectively rather than simply increasing capacity. Controlled experiments demonstrated that LoopLMs yield reasoning processes that align more closely with their final outputs compared to conventional methods. This could reshape the future of AI and machine learning by opening new avenues for scaling and enhancing reasoning among LLMs, indicating a promising direction for next-generation models in the field. The models have been made available for public use, inviting further exploration and development in latent reasoning capabilities.