An Unbiased OSS Benchmark. For Code Review Agents (codereview.withmartian.com)

A new open-source benchmark for evaluating AI code review agents has been established, focusing on tools like CodeRabbit, Copilot, and Gemini Code Assist. This benchmark analyzes thousands of real pull requests (PRs) from GitHub, tracking every AI suggestion, developer interaction, code modification, and resolution thread over the last two months. By leveraging LLM-powered analysis, the benchmark assesses the effectiveness of AI tools based on measurable outcomes, namely their precision, recall, and F1 scores, which quantify how effectively these agents contribute to code quality. The significance of this benchmark lies in its ability to provide a transparent and data-driven evaluation of AI code review systems. By giving developers insight into which AI tools genuinely enhance coding practices and which are less effective, it promotes informed decision-making when selecting tools for code review. Ultimately, this initiative not only aims to improve the reliability of AI contributions in software development but also encourages the continuous refinement of these technologies, fostering a more effective integration of AI in programming workflows.