Mistrial – Re run your LLM judge and see how often it flips (github.com)

The launch of "Mistrial," a new tool for evaluating the reliability of large language model (LLM) judges, addresses the inconsistency issues faced by AI systems in scoring tasks such as evaluation and decision-making. This tool allows users to rerun evaluations across multiple instances of different LLM judges and variations in rubric wording, generating a "reliability card" that quantifies the variability in scores. By enabling developers to assess how often a model changes its verdicts on the same inputs—often due to minor wording changes or the inherent randomness of the models—Mistrial surfaces critical insights about the trustworthiness of LLM evaluations. This development is significant for the AI/ML community as it highlights the need for transparency and reliability in machine-generated judgments, which are increasingly used to gate critical processes. Mistrial provides metrics such as self-agreement rates, Cohen's kappa, and Krippendorff's alpha, allowing teams to identify issues with their evaluators and improve model robustness. The tool also proposes actionable steps when inconsistencies arise, such as flagging unreliable judgments for human review. By addressing this "flakiness" in LLM outputs, Mistrial aims to strengthen the reliability of AI implementations across various applications.