Sakana Fugu (sakana.ai)

An AI agent named Fugu-Ultra has demonstrated significant advancements in optimizing machine learning training recipes by autonomously fine-tuning the parameters of a small GPT model. Utilizing the AutoResearch framework developed by Karpathy et al., Fugu-Ultra executed 123 experiments over approximately 14 hours on a single H100 GPU, achieving an impressive mean bits-per-byte (BPB) score of 0.9774. This performance outpaced three baseline frontier models, showcasing Fugu-Ultra’s capacity to autonomously improve training configurations across various dimensions such as batch size, model depth, learning rates, and optimization settings. The results suggest that orchestrating multiple robust models can surpass the performance of any single leading model in the realm of agentic machine learning research. The implications of this experiment are significant for the AI/ML community, as it highlights the potential for AI agents to independently optimize complex processes, which could streamline research and development in model training. The autonomous discovery of improvements implies a shift towards more efficient and effective ML workflows, potentially accelerating innovations in various applications. Fugu-Ultra not only outperformed existing models in standard benchmarks but also illustrated the agent's ability to adapt and refine code specifications autonomously, paving the way for future advancements in self-improving AI systems.