First Steps Toward Automated AI Research (www.recursive.com)

Recursive has unveiled early results from its automated AI research system, demonstrating state-of-the-art performance across three key benchmarks: fixed-budget language model training, small-model training speed, and GPU kernel optimization. This innovative system automates the entire research loop by proposing experimental ideas, executing them, and using validations to inform future experiments. It is designed to handle multiple research threads simultaneously while retaining insights from prior activities, setting a foundation for recursively self-improving AI. Its performance hinges on optimizing core algorithmic and hardware efficiency levers, making it a significant advancement in automated AI research methodologies. In practical applications, the system showed promise through case studies such as NanoChat Autoresearch and NanoGPT Speedrun. In NanoChat, it outperformed collaborative human-agent efforts, improving a small language model's validation loss from 0.9372 bits per byte (BPB) to 0.9109 BPB in less training time. Additionally, in the highly competitive NanoGPT Speedrun benchmark, the automated system enhanced an already optimized solution, reducing training time from 79.7 seconds to 77.5 seconds while maintaining validation quality. Innovations like FP8 attention projections and novel architectural adjustments highlight the system's capacity to discover significant optimizations, suggesting immense implications for rapid advancements in AI research and deployment.