A General Goal-Conditioned Minecraft Model (pantograph.com)

Pantograph has unveiled a groundbreaking approach to training general robotics models, specifically within the popular game Minecraft, by learning goal-directed behavior through pretraining on vast amounts of internet video data. This innovative method contrasts with traditional techniques that focus on goal-directedness post-training, ultimately enhancing the model's ability to generalize and accomplish complex tasks in diverse environments. The newly developed model, named Pan, features 4 billion parameters and has demonstrated remarkable capabilities, including object exploration, combat, and structure building, while significantly outperforming existing models like STEVE-1 and VLA in various tasks. The significance of this development lies in its potential to revolutionize reinforcement learning by using unlabelled video data as training trajectories to bolster scalability. By treating video frames as goals, the model effectively sidesteps the necessity for explicit reward functions, making the training process more efficient. The initial results showcase Pan’s superior performance in achieving multifaceted objectives, highlighting its adaptability in unforeseen environments. Pantograph plans to further scale these models and extend training across diverse online environments, with aspirations of creating highly capable general robotics systems that can autonomously function across both digital and real-world scenarios. This progress indicates a promising shift towards the development of superhuman AI capabilities in broader domains.