InSight: Self-Guided Skill Acquisition via Steerable VLAs (insight-vla.github.io)

A new framework named InSight has been introduced to enhance the capabilities of vision-language-action (VLA) models in autonomous skill acquisition. Unlike traditional methods that limit learning to predefined demonstrations, InSight allows for the flexible steering of individual primitive actions (e.g., "move gripper to the bowl") within learned skills. This two-stage process uses automated segmentation to break down demonstrations into labeled primitives and a data flywheel that autonomously identifies and acquires missing actions needed for novel tasks. By doing so, InSight enables the robot to learn new skills without any human demonstrations, facilitating a practical foundation for continual learning. The significance of InSight lies in its ability to adaptively acquire and recombine manipulation skills, which is a major advancement for the AI/ML community focused on automation and robotics. The framework has been tested in both simulated and real-world environments through tasks like flipping blocks and pouring liquids, achieving high success rates even with no prior demonstrations of the target actions. By fine-tuning a VLA and using a vision-language model (VLM) to guide the process, InSight enhances efficiency and reduces decision-making time, marking a crucial step toward robots that can autonomously expand their skill sets in dynamic settings.