Harvey LAB-AA: evaluating AI agents on real-world legal work (artificialanalysis.ai)

Harvey has launched the LAB-AA (Legal Agent Benchmark), an innovative evaluation framework for language models specifically designed to tackle real-world legal tasks across 24 practice areas, including corporate law, tax, and litigation. The benchmark assesses models on a set of 120 legal tasks, measuring their performance against a strict rubric of binary criteria. The standout, Claude Fable 5, achieved an all-pass rate of 14.2%, significantly higher than its closest competitors, indicating that while models may successfully meet individual criteria, the complexity of legal work remains a challenge, with 86% of tasks remaining incomplete. This initiative is significant for the AI and machine learning community as it not only highlights advances in legal AI but also provides a structured approach to evaluating language models' effectiveness in nuanced fields. By employing features like context compaction and a new grading method using Gemini 3.1 Pro, Harvey LAB-AA enhances the reliability of outcomes while encouraging further development in AI legal applications. The emergence of tasks and detailed metrics shared on the evaluation page invites deeper engagement from developers and researchers, fostering innovation in creating AI solutions for complex professional environments.