RobotUse (robotuse-team.github.io)

RobotUse has been announced as a groundbreaking platform that bridges the gap between language model decisions and physical robot actions, utilizing a combination of images and geometry. This innovative approach allows the robotic agent to select targets, choose gripper poses, and validate decisions in real-time, as it continuously interacts with its environment. The integration of camera projections and point-cloud views ensures that each decision remains contextual, linking actions directly to visual feedback and the physical scene. This capability is exemplified in multi-task evaluations, where RobotUse successfully completed 54 out of 120 manipulation episodes, achieving a notable 45% task success rate across varying task complexities. The significance of RobotUse lies in its potential for enhancing robotic intelligence through a persistent playbook that learns from previous executions. By retaining reports and observations during task execution, the platform fosters a dynamic learning loop that could significantly improve future robot interactions and task efficiency. The public release includes essential tools such as the agent harness and visual action tools, with a focus on making advancements accessible for developers and researchers. This initiative promises to advance the field of AI/ML in robotics, promoting more sophisticated responses to complex physical tasks while deepening the interplay between language-driven commands and actionable outcomes.