Reap: Automatic Curation of Coding Agent Benchmarks (arxiv.org)

A new paper introduces REAP (Relevance and Execution-Audited Pipeline), an innovative automated curation pipeline designed to create coding agent benchmarks derived from real developer-agent interactions, eliminating the need for manual labeling. Traditional evaluation methods, such as A/B testing and shadow deployment, struggle with speed and reproducibility, leading to unreliable metrics that may not reflect actual productivity. REAP addresses these challenges by implementing an automated verification layer that includes LLM-based task classification, test relevance validation, and multi-run stability checks, ensuring the curated benchmarks provide trustworthy performance signals directly aligned with production usage. The significance of REAP for the AI/ML community lies in its ability to generate practical and reliable benchmarks that more accurately reflect the complexities of coding tasks in production environments. The system has already been employed to curate the Harvest benchmark, which includes real developer prompts and spans over four programming languages, primarily involving Hack. Early evaluations using REAP indicate solve rates between 42.9% and 58.2% across multiple models, highlighting variances in capabilities that can guide critical deployment decisions. This advancement could potentially reshape the landscape of AI coding tools, offering a more effective way to measure and enhance their performance in real-world applications.