Next-Latent Prediction Transformers Learn Compact World Models (2025) (arxiv.org)

A groundbreaking approach in transformer architecture has been unveiled with the introduction of Next-Latent Prediction (NextLat), aimed at improving how these models learn and represent knowledge. Traditional transformers struggle to form compact internal representations due to their reliance on long memory and self-attention without compressing historical data. NextLat addresses this by incorporating self-supervised latent predictions, training the model to derive predictive latent representations based on the next token. This method introduces a recurrent inductive bias while maintaining the efficiency of parallel training and inference. The implications for the AI/ML community are significant, as NextLat shows a marked improvement across multiple benchmarks in world modeling, reasoning, and planning tasks, outperforming standard next-token prediction techniques. By enabling transformers to learn coherent belief states and transition dynamics, this approach facilitates better generalization and representation compression. Notably, it also enhances language modeling inference speed by up to 3.3x through variable-length self-speculative decoding. Overall, NextLat presents a simple yet powerful paradigm that could transform how transformers build compact and predictive world models, offering new avenues for advancements in AI applications.