I audited 162 agent-written PRs – 27% were the AI fixing itself (github.com)

A recent audit of 162 pull requests (PRs) generated by an AI agent revealed that 27% of these submissions were the AI correcting its prior outputs. This innovative tool, Commensa-audit, analyzes git histories from GitHub repositories and offers a detailed report that spans various metrics, including "rework tax," which quantifies the share of PRs that fix earlier work versus those contributing new value. The audit also highlights abandoned attempts, churn clusters, and the persistence of merged code, providing AI/ML engineers with critical insights into the efficiency of AI-generated contributions. The significance of this tool lies in its potential to redefine how AI development is measured, moving beyond traditional metrics of engagement like merged PRs or lines of code shipped. By focusing on the rework ratio, Commensa-audit helps teams identify areas where AI outputs require excessive correction, thereby indicating inefficiencies in AI development processes. As AI agents become more prevalent in software engineering, tools like this may be crucial for improving the overall quality of AI-generated code, guiding teams toward more productive AI-assisted workflows. The self-contained nature of the tool ensures privacy and usability, as it operates locally without sending data over the network.