FML-Bench: A Controlled Study of AI Research Agent Strategies (arxiv.org)

Researchers have introduced FML-Bench, a novel benchmark designed to evaluate AI research agent strategies in machine learning (ML). By featuring 18 fundamental ML tasks across 10 domains, FML-Bench distinguishes agent strategy from execution infrastructure, enabling more accurate assessments of performance driven by different strategies. The benchmark also includes 12 process-level metrics, which analyze exploration behaviors rather than focusing solely on final outcomes. This framework addresses the limitations of existing benchmarks, allowing for deeper insights into how different strategies perform across various contexts. The findings from FML-Bench reveal that a simple greedy hill-climber can perform comparably to more complex tree-search agents, suggesting that strategy simplicity can sometimes yield strong results, especially in environments with dense improvement opportunities. Additionally, a newly designed adaptive agent that shifts its exploration tactics based on performance stagnation demonstrated superior results, pointing to the importance of flexibility in strategy. Early convergence and focused exploration emerged as critical factors tied to the final performance, while aspects like solution diversity and compute costs were less impactful, offering the AI/ML community valuable principles for enhancing research agent efficiency.