We spent $15k and benchmarked 5 DOCX-editing approaches on Harvey LAB (www.vespper.com)

A recent evaluation of five DOCX-editing approaches using Harvey LAB, an open-source Legal Agent Benchmark, highlights the state of AI-powered document editing, especially relevant to industries reliant on Word documents like legal and finance. The study compared tools—including baseline methods using python-docx and MCPs (Managed Code Providers)—to assess efficiency, cost, and pass rates across 180 tasks summing up to 2,010 tasks in 27 practice areas. Results showed that the baseline approach yielded the highest mean rubric score and finished all tasks at significantly lower costs, while some MCPs struggled with context window limits and achieved lower completion rates. The findings underscore a critical insight: while MCPs are designed to reduce cognitive load, they often led to higher token consumption and slower performance compared to traditional methods. The editing phase dominated overall processing, consuming 85% to 95% of the input tokens, highlighting the resource-intensive nature of working with DOCX files. These results are pivotal for the AI/ML community, showcasing the ongoing challenges in automating complex document editing tasks and offering insights into optimizing AI agent frameworks in real-world applications.