Claude as a Robot Teacher (irvin.quest)

A recent project has successfully developed Claude as a robot teacher by leveraging an innovative two-stage approach. Initially, a Claude-based controller predicted outcomes for various robotic actions, but due to slow performance, the researcher switched to an autonomous learner that combines data collection and model training in a unified process. This system uses Claude to analyze various states—data collection, validation, training, and more—deciding the sequence of actions based on the data provided. Although this integration streamlined the workflow, Claude's decision-making didn’t significantly enhance model performance, revealing room for improvement in managing human interventions. The final model, a diffusion transformer with around 1 billion parameters, demonstrated adequate predictive capabilities but struggled with live inference. To enhance efficiency, the researcher transitioned to behavior cloning, training a smaller image-to-action classifier (ResNet18) that drastically reduced action response time from 25 seconds to just one second. This outcome suggests that a well-tuned simpler model could effectively learn from a less sophisticated world model, potentially reshaping training methodologies in robotics. Future directions include advancing the robot’s ability for complex tasks like pick-and-place operations and enabling it to generalize movements across multiple joints.