Nvidia Cosmos Policy for Advanced Robot Control (huggingface.co)

Nvidia has announced the launch of Cosmos Policy, a groundbreaking advancement in robot control and planning that builds on its Cosmos world foundation models (WFMs). This new policy demonstrates state-of-the-art performance in manipulation tasks by post-training the Cosmos Predict-2 model to encode robot actions and future states as latent frames—essentially treating this data like video frames. This innovative approach allows for a unified model capable of predicting action chunks, future robot observations, and expected returns, all while leveraging a single architecture trained using diffusion processes over continuous spatiotemporal latents. The significance of Cosmos Policy for the AI/ML community lies in its superior performance on established benchmarks like LIBERO and RoboCasa. It outperforms existing diffusion and vision-language-action models, achieving remarkable success rates, which enhances generalization and efficiency in robotic tasks. Additionally, Cosmos Policy's unique structure supports flexible deployment as either direct or planning policies, improving task completion rates in complex scenarios. Nvidia is further promoting engagement in this field through the Cosmos Cookoff, an open hackathon designed for developers to explore and innovate with these advanced models, thereby fostering collaboration and advancing practical applications in robotics and autonomous technology.