Every AI code review vendor benchmarks itself, and wins (deepsource.com)

Recent announcements from various AI code review vendors have highlighted a significant issue: the lack of a standardized benchmark for evaluating AI tools in code review. Unlike coding agents that have a shared benchmarking framework (SWE-bench), AI code review tools currently rely on individual vendors' self-defined metrics, making comparisons between them virtually impossible. This lack of a consistent measurement hinders engineering leaders from making informed decisions, often forcing them to rely on subjective evaluations and demos rather than objective performance data. The implications of this disparity are profound for the AI/ML community. Without standardized benchmarks, vendors can publish inflated performance metrics that may lack scientific rigor, with significant variations in reported results based on who conducts the evaluations. For example, Greptile and Augment Code both evaluated the same repositories yet reported drastically different F1 scores due to subjective interpretations of what constitutes a bug. Experts stress the need for independent evaluations, real-world datasets, and reproducible methodologies to create reliable benchmarks. Until a community-maintained standard emerges, users must approach vendor benchmarks—whether their own or those of others—with a healthy skepticism, particularly as they often reflect optimized outcomes rather than true capability.