I built a free tool to evaluate AI agent outputs (human labels and LLM judges) (github.com)

A new open-source tool called Verdict has been launched, designed to help developers evaluate the outputs of AI agents, chatbots, and other machine learning features. The tool allows users to load conversation traces, label them as Pass or Fail, and perform comprehensive error analysis using grounded theory methods. Key steps include tagging failures, clustering these into categories, and building a machine learning judge that can autonomously evaluate future outputs based on the validated human labels. The entire process is executed in the browser without requiring backend support, making it accessible and preserving user privacy. This tool is significant for the AI/ML community as it addresses the critical need for robust evaluation measures that go beyond simple qualitative assessments (often referred to as a "vibe check"). By implementing a structured evaluation loop, Verdict allows developers to identify failure modes systematically and automate quality monitoring in production. With functionalities such as anonymization, local execution, and a thorough validation process for the LLM judges, Verdict empowers users to ensure AI features operate effectively and improve over time, ultimately fostering more reliable AI applications.