Benchmarking the Thomson Reuters legal agent (www.thomsonreuters.com)

Thomson Reuters detailed how it benchmarks and hardens CoCounsel, its GenAI legal assistant, by combining attorney subject-matter expertise with large-scale automated testing via Scorecard — an evaluation platform built by engineers who created Waymo’s self-driving test infrastructure. Scorecard runs millions of agent simulations, compares model outputs to gold‑standard attorney-crafted responses, and assigns numeric pass/fail scores on recall, precision and accuracy. The post highlights a concrete migration case: when the Review Documents skill moved to a new underlying model, Scorecard flagged a failing testcase (identifying a patient’s current medications). Attorney reviewers plus engineers iteratively adjusted backend prompts and recalibrated the skill in staging until scores rose from 1/5 to 4/5 and consistency reached near 100%, after which continuous daily Scorecard checks continued in production. For the AI/ML community this is a practical blueprint for operationalizing reliability in domain-critical agents: it shows the need to treat each skill as a model-calibrated tool (so migrations require retuning), separate failures caused by input quality versus system limitations, and combine curated testsets, “least-common-denominator” testcases, human gold standards, and automated scoring to accelerate debugging and validate fixes. The approach underscores best practices for safe deployment—staging, iterative prompt engineering, continuous monitoring, and SME-in-the-loop evaluation—relevant to legaltech, fintech, compliance and any high-stakes AI application.