AI coding agent PR merge rates: Claude 84%, Codex 74%, Devin 43%, humans 85% (arxiv.org)

A recent empirical study has revealed significant insights into the performance of AI coding agents in software development, specifically analyzing their pull request (PR) merge rates. The study found that leading AI models, Claude, Codex, and Devin, demonstrate merge rates of 84%, 74%, and 43% respectively, compared to an 85% merge rate by human developers. By examining the AIDev dataset, researchers assessed how these AI contributions vary across different stages of the software development lifecycle, providing a nuanced understanding of their effectiveness relative to human-generated PRs. The significance of this study lies in its comprehensive analysis of the evolving role of AI in software engineering, highlighting both the benefits and limitations of AI agents in enhancing software quality. By identifying the development tasks where AI is most effectively applied and observing the characteristics of AI-generated versus human-generated PRs over time, the study not only sheds light on the current capabilities of AI coding agents but also encourages further exploration of their integration into development workflows. This research contributes to the broader discourse on AI in software engineering and aims to refine how these tools can be leveraged to complement and enhance human efforts in code quality and efficiency.