LLMs Don't Quite Beat Classical Hyperparameter Optimization Algorithms (github.com)

Recent research has shown that large language models (LLMs) fall short of surpassing classical hyperparameter optimization (HPO) algorithms. The study evaluated LLMs’ capabilities in optimizing hyperparameters for a small language model against established methods like CMA-ES and TPE, particularly under fixed compute budgets. The results indicated that classical methods consistently outperformed LLM-based approaches, especially due to their better management of out-of-memory (OOM) failures—crucial for sustaining multiple optimization trials. While allowing LLMs to edit training code directly narrowed this gap, it didn’t fully bridge it, with limitations in the LLM’s ability to track optimization states across trials. To leverage the strengths of both techniques, the researchers introduced a hybrid approach named Centaur, which combines CMA-ES with LLM output. This method showed promising results, outperforming all other configurations tested, including traditional and purely LLM-based methods. Notably, a smaller 0.8B LLM proved sufficient to exceed the performance of classical algorithms when integrated with CMA-ES. The findings underscore that while LLMs can enhance HPO processes, they are more effective as complementary tools rather than replacements for classical optimization methods, suggesting a future avenue for hybrid applications in the AI/ML community.