Using evals to cut AI errors 7x (hex.tech)

Hex recently announced a significant improvement in AI feature quality by utilizing a novel method called "hill climbing" through the use of 1,800 evals, which reduced their AI model's error rate from 21% to just 3%. By focusing on refining their feature development process from the outset rather than treating evals as a final quality assurance step, Hex was able to enhance the accuracy of their Generative Apps. This new approach emphasized the role of a robust validation mechanism over traditional prompt optimization, revealing that adjustments to the validator accounted for over half of the accuracy gains. The impact of this advancement is particularly important for the AI/ML community as it showcases a more dynamic integration of evals into the development lifecycle, enabling quicker iterations and higher reliability in AI outputs. By designing eval cases rooted in real user requests and focusing on critical metrics that prioritize reducing wrong edits, Hex not only improved user trust in automated chart editing but also laid the groundwork for future innovations in AI-driven applications. This case serves as a valuable lesson for developers across the industry on the importance of optimizing the entire feature ecosystem beyond just model prompting.