Evaluating Gemini Robotics Policies in a Veo World Simulator (veo-robotics.github.io)

Gemini Robotics has introduced a groundbreaking generative evaluation system based on their frontier video model, Veo, significantly enhancing the policy evaluation landscape in robotics. This system allows for comprehensive assessments of robotic policies across various scenarios, beyond traditional in-distribution evaluations, effectively covering out-of-distribution (OOD) generalization, safety assessments, and more. With a robust framework for simulating realistic environments through advanced generative image-editing and multi-view video generation, the Veo model captures the complexities of real-world interactions, enabling precise predictions of policy performance across diverse tasks. The implications of this development are substantial for the AI/ML community, as it facilitates a deeper understanding of the factors affecting robotic performance, including the influences of novel objects, visual backgrounds, and potential distractors. By conducting over 1,600 real-world evaluations involving Gemini Robotics' policy checkpoints, the system not only ranks policy effectiveness but also identifies vulnerabilities in critical tasks through predictive red teaming. This innovative approach positions Veo as a pivotal tool in advancing safety and generalization in robotic applications, marking a significant step forward in integrating generative models for enhanced robotic capabilities.