Why You Can't Trust Most AI Studies (www.thealgorithmicbridge.com)

Two recent high-profile studies landed on opposite sides of the generative-AI debate: an MIT analysis (150 interviews, a 350-employee survey, and review of ~300 public deployments) concluded about 95% of AI pilots “fail” when judged by short-term P&L, while a Wharton report claims roughly 75% of enterprises are already seeing positive ROI. Those headlines are both true to their narrow definitions but misleading if taken at face value. MIT’s metric focuses on early-stage pilots and immediate profit impact, and flags a “learning gap” in organizational integration; it also suffers from visibility bias because public, flashy pilots are overrepresented. The Wharton finding likely reflects more mature deployments and different ROI definitions (and may suffer survivorship or selection bias). For the AI/ML community these studies are a useful case study in why single-study narratives are fragile: wartime incentives push extremes, publication and visibility biases magnify striking results, and choice of metric (short-term P&L vs. productivity, time-saved, error rates, workflow quality) radically changes conclusions. Practitioners and researchers should scrutinize sample selection, time horizon, and outcome definitions, prioritize replication and granular operational metrics, and treat bold headlines as signals to dig into methodology rather than as verdicts on the technology’s value.