The gauge broke: devs felt 20% faster with AI, measured 19% slower (intrepidkarthi.com)

A recent controlled trial by METR has revealed a surprising disconnect between perceived and actual productivity among experienced developers using AI tools. While participants felt that AI made them approximately 20% faster, the stopwatch measured their performance as nearly 19% slower. This discrepancy arises because AI, while expediting certain tasks like typing, introduces significant overhead in prompting, waiting, and reviewing outputs that can often be subtly wrong. As a result, experienced developers working with familiar codebases might experience a slowdown during the most cost-intensive stages of their work—review and verification. This finding holds significant implications for the AI/ML community, particularly for product managers and engineering leaders making decisions based on perceived productivity metrics. The study indicates that the developers most confident in the AI's speed-enhancing capabilities were often those experiencing measurable slowdowns. Moreover, telemetric data from platforms like DORA and GitClear shows trends of increased merge rates and code churn without a corresponding increase in delivered value, undermining the initial optimism surrounding AI integration. To navigate this landscape effectively, teams are advised to rely less on subjective feelings of speed and instead focus on quantifiable outcomes in production, highlighting the need to reassess productivity metrics in light of these findings.