Measuring Impact of Early-2025 AI on Experienced Open-Source Dev Productivity (arxiv.org)

Researchers ran a randomized controlled trial (Feb–Jun 2025 tools) to measure how contemporary AI coding tools actually affect experienced open-source developer productivity. Sixteen developers (median ~5 years on each project) completed 246 real tasks in mature repositories, with each task randomly assigned to either allow or disallow use of early‑2025 AI tools (primarily Cursor Pro and Anthropic’s Claude 3.5/3.7 Sonnet). Beforehand developers forecast a 24% speedup with AI and afterward estimated a 20% speedup, but the measured effect was the opposite: AI access increased task completion time by 19%. This slowdown also contradicts expert forecasts from economics (39% faster) and ML (38% faster). The study’s randomized design gives causal weight to the surprising result and forces the community to reconsider assumptions about out‑of‑the‑box productivity wins. Authors evaluated 20 contextual factors (project size, code quality standards, prior AI experience, etc.) and found the slowdown robust across analyses, making experimental artifacts an unlikely sole explanation. For AI/ML practitioners and tool builders, implications include: the need to reduce integration and cognitive overhead, improve reliability/hallucination handling, and design evaluations in ecologically valid settings. The paper highlights that measurable, real‑world productivity gains are not guaranteed even with advanced models and underscores the importance of rigorous field trials.