Tech companies measure the impact of AI on software development (newsletter.pragmaticengineer.com)

Pragmatic Engineer’s latest deep dive, led by CTO Laura Tacho of DX, aggregates how 18 tech firms (including Google, GitHub, Microsoft, Dropbox, Monzo and Atlassian) measure the real-world impact of AI on software development. The piece highlights a widespread “measurement gap”: execs want headline gains (lines of code, claimed shipping percentages) but lack rigorous metrics to judge cost, quality and long-term maintainability. DX’s AI Measurement Framework — built from analysis of 400+ companies — argues you don’t need wholly new KPIs, but you do need AI-specific signals layered on solid engineering baselines. Practically, companies combine core engineering metrics (Change Failure Rate, PR throughput, PR cycle time, developer experience) with AI-specific ones (adoption: DAU/WAU, CSAT for AI tools, time saved per engineer, AI spend). Firms use cohort and before/after analyses, compare AI vs non-AI users, and slice by role, tenure and language to spot winners and risk areas. Examples: Dropbox reports 90% weekly AI adoption with ~20% higher PR merges and reduced change-failure rate for AI users; Webflow sees ~20% PR throughput gains for longer-tenured devs. The article warns of vendor data hoarding, hidden tech debt from generated code, and recommends baseline measurements, an experimental mindset, and paired metrics that guard quality while tracking velocity.