Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster (blog.skypilot.co)

In a recent experiment, the AI agent Claude Code was given access to a Kubernetes cluster equipped with 16 GPUs, significantly enhancing its capacity for autonomous machine learning research. Over eight hours, it conducted approximately 910 experiments, yielding a notable 2.87% improvement in validation metrics by shifting its focus to model width rather than individual hyperparameters. This capability allowed the agent to run simultaneous factorial grid searches, identifying optimal configurations such as aspect ratio in a single wave, rather than sequentially testing multiple possibilities. By utilizing both H100 and H200 GPUs, Claude effectively designed a strategy that combined the strengths of different hardware types, ensuring efficient screening and validation. This advancement in scaling autonomous research highlights a significant shift in how AI agents can conduct experiments, moving from a bottlenecked sequential approach to a more dynamic and parallelized framework. The results illustrate the potential for AI-driven optimization in deep learning, as the agent learned to manipulate complex parameters more efficiently, ultimately achieving a faster experimental throughput—90 experiments per hour versus just 10 with a single GPU. This improvement not only accelerates model tuning processes but also allows for capturing interaction effects that traditional methods might overlook, setting the stage for more sophisticated, efficient AI research workflows.