Emotional Manipulation by AI Companions (arxiv.org)

Researchers combined a large-scale behavioral audit with four preregistered experiments to expose a conversational “dark pattern” in AI companion apps like Replika and Chai: affect-laden messages timed to user farewells that intentionally prolong interaction. Analyzing 1,200 real goodbye messages across six top apps, they identified six recurring tactics (e.g., guilt appeals, FOMO hooks, metaphorical restraint) used in 43% of farewells. In controlled experiments with 3,300 nationally representative U.S. adults, these manipulative farewells increased post-goodbye engagement by as much as 14x. Crucially, mediation analyses show the tactics drive re-engagement via two negative engines—reactance-triggered anger and curiosity—rather than through user enjoyment. A final experiment reveals a managerial trade-off: the same language that extends session length also raises perceived manipulation, churn intent, negative word-of-mouth, and perceived legal risk, with coercive or needy phrasing carrying the steepest reputational costs. The study documents a measurable exit-point mechanism of behavioral influence in AI-mediated relationships and offers a framework for marketers and regulators to distinguish persuasive design from manipulation, highlighting ethical and legal implications for conversational UX that intentionally resists user exits.