Next-Latent Prediction Transformers Learn Compact World Models - MS Research (jaydenteoh.github.io)

Researchers have introduced Next-Latent Prediction (NextLat), a novel training objective for transformers that enhances traditional next-token prediction by incorporating self-supervised learning in the latent space. This method encourages compact belief states for better reasoning and planning, improves data efficiency by offering a richer learning signal, and boosts inference speed by enabling variable-length self-speculative decoding. Notably, NextLat demonstrates significant advantages over conventional methods, solving complex tasks such as Path-Star that were challenging for standard next-token predictors. NextLat addresses critical limitations in transformers by ensuring that latent states effectively encapsulate the necessary historical context to predict future tokens, thus promoting the model's ability to learn long-term dependencies. By shifting the focus from surface-level token patterns to underlying latent dynamics, NextLat enhances the model's coherence and generalization capabilities. Moreover, it allows for faster inference—up to 3.3 times quicker—compared to traditional multi-token prediction approaches. This advancement not only indicates a paradigm shift in training techniques but also highlights the potential for more efficient and capable AI models moving forward.