Non-Obvious Things I Learned About GEPA (www.elicited.blog)

In a recent exploration of the Genetic-Pareto (GEPA) optimizer, a developer shared valuable insights gained from a trial run that initially resulted in disappointing outcomes. GEPA, designed to evolve prompts by leveraging a large language model (LLM) to critique failures and propose improvements, distinguishes itself by maintaining multiple candidate pools—allowing candidates to thrive based on their performance across different validation examples, rather than average performance. This key feature ensures that even if a candidate excels in a specific area but is mediocre overall, it still has the potential to influence future generations of prompts. The learnings underscore the importance of two main factors: the selection of validation examples and the strategic design of the metric for scoring candidates. A well-curated validation set promotes diverse input representation, enabling the retention of specialists that can tackle varying challenges. Additionally, to accommodate multi-objective optimization, developers are encouraged to integrate their preferences directly into the scoring function. By balancing exploration and exploitation and adjusting strategies based on frontier dynamics, GEPA offers a nuanced approach to optimizing artificial intelligence prompts, making it a valuable tool for the AI/ML community.