Multimodal learning with next-token prediction for large multimodal models (www.nature.com)

A groundbreaking advancement in multimodal learning has been announced with the introduction of Emu3, a family of models that applies next-token prediction to unify the generation of text, images, and video. This approach eliminates the reliance on diffusion models or compositional systems, previously dominant in the field, by standardizing multimodal data into a single token stream. Emu3 demonstrates competitive performance against specialized models in both perception and generation tasks, showcasing its capabilities in high-fidelity video generation and vision-language-action modeling for robotics. The significance of Emu3 lies in its potential to simplify multimodal learning, making it more scalable and efficient by focusing solely on next-token prediction. The model's ability to generate and understand multimodal content using a unified architecture may pave the way for the development of more advanced artificial intelligence systems. Additionally, the open-sourcing of key techniques, including a novel vision tokenizer and extensive training methodologies, will encourage further exploration and innovation in multimodal models, marking a pivotal step toward achieving unified multimodal intelligence.