The Honest Raw LLM Code Review Benchmark: 21 Models, 7,980 Reviews (ironbee.ai)

The recent launch of "The Honest Raw LLM Code Review Benchmark" marks a significant advancement in evaluating the capabilities of language models in code review tasks. Researchers analyzed 21 different AI models across 7,980 reviews of real pull requests, measuring performance at varying effort levels. This extensive benchmark assesses models based solely on their ability to review code without modification from external tools or prompts. The results reveal that the best-performing raw models are on par with the leading existing code review tools, showcasing the models' inherent proficiency in identifying issues in code. This benchmark is crucial for the AI/ML community as it offers the most comprehensive public evaluation of LLM capabilities in the context of code reviews to date. The findings emphasize that higher costs and advanced versions do not always equate to better performance, as some older or less expensive models achieved competitive scores. Notably, cheaper options like Muse Spark 1.3 and Claude Sonnet 5.5 provide impressive performance, challenging assumptions about the relationship between cost, model sophistication, and effectiveness. By making all data and methodologies publicly available, the benchmark supports transparency and further innovation in AI-driven code analysis tools.