We built a real-world benchmark for AI code review (www.qodo.ai)

Qodo's research team has launched the Qodo Code Review Benchmark 1.0, a groundbreaking tool designed to rigorously assess the efficacy of AI-powered code review systems, including their own Qodo Git Code Review. This benchmark addresses the limitations of existing evaluation methods, which mostly focus on isolated bug detection and fail to encompass a comprehensive view of code quality and best-practice adherence. Instead of relying on backtracking from fix commits, Qodo's approach injects defects into genuine, merged pull requests (PRs) from active open-source projects, allowing for a robust evaluation of both code correctness and quality in a realistic context. The benchmark evaluates 100 PRs containing 580 defects, significantly enhancing the scale and scope of testing compared to previous benchmarks. In comparative evaluations against seven leading AI code review tools, Qodo demonstrated superior performance with an impressive F1 score of 60.1%, capturing a broader range of issues while maintaining high precision. This new standard not only provides a scalable and repository-agnostic mechanism for generating high-quality evaluation data but also ensures that AI tools can better address the complexities of real-world code reviews, reflecting the diverse challenges developers encounter. The benchmark is now publicly available on GitHub, promoting transparency and further research in advancing AI-driven code review capabilities.