Show HN: GEDD – Find what your AI agent gets wrong (before your users do) (github.com)

GEDD has been launched as a novel tool to evaluate AI agents before they face end-users, addressing critical gaps in traditional evaluation methods. It allows a domain expert to engage in a conversation, producing a production evaluation pipeline within 90 minutes. This pipeline focuses on generating "golden queries" and assessing an agent’s performance against real-world contexts, significantly improving the identification of potential failures that may not be captured by pre-existing rubrics. This tool is particularly significant for the AI/ML community as it enables a continuous feedback loop, where production failures inform new test cases, allowing the evaluation framework to evolve alongside the agent. GEDD's methodology emphasizes the importance of contextual understanding—errors are framed using domain-specific language, making the feedback relevant and actionable. By integrating GEDD with AWS native services such as SageMaker, the evaluation process remains smooth and efficient. Overall, GEDD redefines how AI agent evaluations are conducted, ensuring that nuanced failures can be caught early, ultimately enhancing the reliability of AI systems.