A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation (arxiv.org)

GroundEval has been introduced as a groundbreaking framework for evaluating AI agents by moving away from the reliance on Large Language Model (LLM) judges. This deterministic testing method systematically assesses whether agents utilized the correct evidence in their decision-making processes. A contrasting case study highlighted a discrepancy where two LLM judges rated an agent's response favorably, but GroundEval revealed a critical flaw: the agent provided an answer without ever retrieving the necessary information, resulting in a score of 0.000. This framework addresses fundamental failures that LLM judges often miss, particularly around evidence validity and reasoning accuracy. GroundEval creates a structured environment for assessing agent responses based on grounded, time-bounded, and access-controlled evidence. Its evaluation focuses on three key areas: whether agents checked for necessary evidence, if they reasoned based only on what was available at the moment, and whether they used correct causal mechanisms. By providing detailed diagnostics for each question and making evaluation results inspectable, GroundEval enhances reliability in evaluating AI agent performance, marking a significant step forward for the AI/ML community in ensuring accountability and transparency in agent behavior.