Frontier models score under 2/100 controlling a simulated humanoid (www.vals.ai)

Recent evaluations of frontier models in robotic control revealed startlingly low performances, with none scoring above 2 out of 100 on humanoid tasks. The study tested various AI models, including GPT and Claude series, across 24 tasks involving a 7-DoF robotic arm and a 29-DoF humanoid. While the GPT-6.1 Sol model managed a score of 51 on arm tasks, it, along with other models, completely failed to navigate the complexities of humanoid tasks, illustrating a significant gap in current AI capabilities for complex robotic operations that demand integrated perception and real-time response. The significance of these findings underscores the challenges AI researchers face in developing general-purpose agents that can effectively control robotic systems. The evaluation criteria drew from realistic robotic task parameters, requiring models to process sensory data and execute motor commands reminiscent of real-world scenarios. The low scores indicate notable deficiencies in decision-making speed and context understanding, highlighting the need for improved learning frameworks that can enhance planning and execution under realistic constraints, particularly in dynamic environments. This study not only identifies the current limitations of leading AI models but also sets a crucial benchmark for future advancements in AI-driven robotics.