We Are Changing Our Developer Productivity Experiment Design (metr.org)

METR has announced changes to its developer productivity experiment initially launched in early 2025, which previously indicated that AI tools slowed experienced open-source developers by 20%. In a new study beginning in August 2025 with a larger participant pool and various AI tools, researchers found that increased reliance on AI has skewed the results, as many developers opted out due to preference for AI assistance. This resulted in a biased sample, likely underestimating the potential productivity gains from AI, as those who still chose to participate may represent less enthusiastic AI users. The initial findings suggested a speedup effect varying from -4% to -18%, but the true impact could be significantly positive if considering the omitted developers who favor AI. The significance of this study lies in its potential implications for understanding how AI adoption impacts developer productivity. Researchers plan to address the selection issues through more intense experiments and observational studies, as well as exploring different methodologies such as fixed-task designs and evaluating AI's capabilities to perform tasks autonomously. The findings reflect a broader trend in the tech community where reliance on AI is rapidly changing task dynamics, emphasizing the need for continual adaptation in research methodologies to capture accurate productivity measurements in an evolving landscape.