Can LLMs Beat Classical Hyperparameter Optimization Algorithms? (arxiv.org)

A recent study on autoresearch has explored the capabilities of large language models (LLMs) in hyperparameter optimization (HPO) compared to classical algorithms such as CMA-ES and TPE. The research found that while classical methods consistently outperformed LLM-based approaches in tuning hyperparameters of a small language model, allowing LLMs to edit training code helped reduce the performance gap. Despite using advanced LLMs like Claude Opus 4.6 and Gemini 3.1 Pro Preview, challenges remained in tracking optimization state across trials, highlighting the strengths and weaknesses of each approach. To capitalize on the advantages of both methodologies, the researchers developed a hybrid system called Centaur, which integrates the interpretable states of classical methods with LLM capabilities. In their experiments, Centaur demonstrated superior performance, with even a smaller 0.8B LLM surpassing all classical and pure LLM techniques. The study underscores the potential of LLMs as complementary tools rather than replacements for classical optimization methods, emphasizing the need for larger models in unconstrained scenarios to compete effectively.