CAP-X: LMs' First Physical Exam (capgym.github.io)

A groundbreaking development in AI was announced with the introduction of CaP-Bench, a comprehensive benchmarking platform designed to evaluate how well large language model (LM) agents can write executable code for robotic control tasks. This initiative is significant as it bridges the gap between advanced language models and practical applications in robotics, demonstrating that today's state-of-the-art models can generate effective robot control policies from natural language instructions, achieving over 30% success without any task-specific training. This marks a shift in the belief that only specialized models are capable of complex tasks in manipulation. The CaP-Agent0, a training-free coding agent, exemplifies this potential by achieving an 18% success rate in manipulating tasks, while the application of reinforcement learning through CaP-RL significantly boosts performance, with a model improving from 20% to 72% success in simulation. This remarkable progress is further evidenced by the model's ability to transfer learned policies to a real robot, nearing human expert performance in cube manipulation tasks. The findings indicate a promising approach where lightweight language models can effectively pair high-level planning with visual-motor policies, suggesting new pathways to enhance the capabilities of even smaller AI models in robotic applications.