Nvidia Cosmos 3 (developer.nvidia.com)

NVIDIA has announced the release of Cosmos 3, a groundbreaking frontier foundation model for physical AI that integrates physical reasoning, world generation, and action generation into a unified open-source framework. This model is designed to enable robots, autonomous vehicles, and smart spaces to comprehend their real-world environments, make predictions, and generate appropriate actions. The Cosmos 3 model features a Mixture-of-Transformers architecture, comprising a Reasoner tower, which is a vision-language model that interprets multimodal observations, and a Generator tower that produces future actions and observations based on the Reasoner's input. By combining these functionalities, Cosmos 3 simplifies development and enhances efficiency. The significance of Cosmos 3 lies in its potential to empower a new wave of physical AI applications; it is now available in two variants: Cosmos 3 Nano, optimized for real-time inference on workstations, and Cosmos 3 Super, a high-capacity model aimed at large-scale deployments. Both models are equipped with a suite of open datasets and post-training scripts to facilitate domain adaptation for various tasks, from robotics to autonomous driving. Additionally, NVIDIA's new Human Evaluation (HUE) framework offers a more objective assessment of video generation quality, enhancing comparison across models. With its open-source nature and optimized deployment capabilities, Cosmos 3 promises to accelerate development in the rapidly evolving field of AI and machine learning.