Use LLM for direct robot control (arxiv.org)

Recent research has unveiled the potential of large multimodal models like GPT-6 Astra for direct robot control, highlighting significant advancements in robotic manipulation capabilities. This study evaluates the model's performance in closed-loop manipulation tasks, assessing its abilities across key areas such as depth perception, in-context learning, and tactile feedback. Notably, GPT-6 Astra demonstrated success rates of 26.92% on RoboDojo and 76.0% on RoboTwin 2.0, indicating competitive performance while also revealing a significant gap between semantic reasoning and physical execution, particularly in tasks involving complex motion coordination and consequence-aware action planning. The introduction of the LIBERO-GeoMotion benchmark showcases the challenges faced when translating spatial understanding into feasible robotic motion, as the model achieved only 11.1% zero-shot success in geometry-constrained tasks. Real-world trials further underline persistent deficits in motion feasibility, precise control, and safety—hurdles exacerbated by substantial inference latency. This research not only characterizes the emerging capabilities of general-purpose multimodal models in robotics but also emphasizes the critical issues that need to be addressed for their reliable and efficient deployment in practical applications, marking a pivotal step toward enhancing robotic functionality and safety in everyday tasks.