Automated Discovery Has No Universally Superior Harness (arxiv.org)

A recent study highlights the limitations of current autonomous discovery systems like OpenEvolve and TTT-Discover, revealing that there is no universally superior harness for various machine learning problems. By analyzing over 3.1 million rollouts across 30 budget-matched harnesses, the researchers concluded that harness choice should be treated as a hyperparameter tailored to specific problem contexts rather than a one-size-fits-all solution. This challenges the prevailing belief that complex harnesses are inherently superior, as simpler alternatives often outperform them in practical applications. The findings also underscore the significance of early discovery performance as a predictor of final outcomes, leading to the development of an adaptive-allocation strategy that reallocates computational resources from weaker to stronger harness runs. This approach not only enhances efficiency but also suggests a crucial shift in focus from static harness selection to dynamic adaptation based on real-time performance metrics. By releasing comprehensive datasets for future research, the study aims to pave the way for better tools in the AI/ML field, fostering a more refined and empirically grounded understanding of discovery harnesses.