Vibes vs. Evidence: What delivers AI code review quality (dsifry.github.io)

A recent study has evaluated the effectiveness of different AI code review setups in discovering real bugs and their associated costs and processing times. Eight AI models were tested across a range of conditions—using single-prompt reviews and more sophisticated open-source harnesses like Compound Engineering and metareview. The findings revealed that the harness setups consistently outperformed one-shot prompts in identifying bugs, achieving a mean increase of 13.5 percentage points in recall. However, this higher performance came at a cost, requiring significantly more tokens, dollars, and time per review. For instance, while a single prompt review averaged 54 seconds, harness reviews took nearly 2.8 minutes and incurred higher costs. The study emphasizes the benefits of using open-weight models in conjunction with low-effort harnesses for optimal value. Specifically, GLM-5.3 running the metareview harness at low effort was found to be the most cost-effective, identifying 72 bugs for just $0.22 per review. The results suggest that adopting a harness method not only enhances bug identification but also calls for a strategic choice of models to manage costs effectively, making it crucial for developers and teams engaged in AI-enhanced code reviews to weigh these factors in their workflows.