Auto-autoresearch: self-improving agents on Karpathy's NanoChat benchmark (rekursiv.ai)

rekursiv.ai has introduced "Auto-autoresearch," a novel approach in which AI agents autonomously improve language models within a constrained training budget, specifically focusing on Karpathy's NanoChat benchmark. In a striking feat, these agents executed over 6,164 experiments in just three days, experimenting with diverse improvements including training data, architecture, tokenizers, and custom triton kernels. Their collaborative design not only optimized the model, reaching a mean bits per byte (BPB) score of 0.887791 but also enhanced the research process by creating a system that learned from previous experiments, revised instructions, and refined its internal communication. This development holds significant implications for the AI/ML community as it showcases the potential for self-improving AI systems to identify strategies for effective research conduct autonomously. Enabled by Trackinizer, an open-source custom graph database, the system documented the lineage of hypotheses, experiments, and results, establishing a knowledge base that future researchers can leverage. The iterative research campaigns underscored how adaptable AI can drive rapid advancements in model performance while simultaneously refining the research methodology, potentially accelerating the pace of discovery in the field.