NCP-ArchPreview: 8.9B latent LM matches OLMo-3-7B on 51% of tokens (arxiv.org)

Researchers have introduced NCP-ArchPreview, an innovative latent-space language model that enhances autoregressive pretraining by integrating Next Concept Prediction (NCP) alongside traditional next-token prediction (NTP). This groundbreaking approach enables the model to predict multi-token concepts, significantly elevating the complexity of language generation tasks. With 8.9 billion parameters trained on a massive 5.73 trillion tokens from the Dolma-3 dataset, NCP-ArchPreview has achieved a remarkable performance milestone by matching the pretraining loss of OLMo-3-7B using only 51.3% of the training tokens. Additionally, it outperforms OLMo-3-7B by 2.45 points in downstream tasks, with a notable 5.99-point gain on the GSM8K benchmark. The significance of this development lies in its potential to reshape how language models understand and generate text by leveraging a learned latent space for more efficient training and performance. Not only does the architecture reduce computation to just 85% of that required by similar models, but it also facilitates domain adaptation through a lightweight interface created by updating a small 17 million-parameter VQ module. The ability to inject concept representations into other applications, such as the DFlash2 drafter, further illustrates the practical implications of this model, suggesting that NCP-ArchPreview could drive significant advancements in future AI applications.