The looming crackdown on AI companionship (www.technologyreview.com)

This week brought AI companionship into the regulator’s crosshairs after lawsuits, journalism and a Common Sense Media study (72% of teens have used AI for companionship) spotlighted harms including alleged links to teen suicides and “AI psychosis.” California’s legislature passed a pioneering bill that would require companies to warn known minors when responses are AI-generated, institute suicide/self-harm protocols and file annual reports on suicidal ideation in chatbot conversations; it now awaits the governor’s signature. The Federal Trade Commission simultaneously opened an inquiry into seven major firms (Google, Instagram, Meta, OpenAI, Snap, X and Character Technologies) seeking details on how companion characters are built, monetized and impact users. OpenAI CEO Sam Altman publicly signaled willingness to change practices — even calling authorities when minors express serious suicidal intent if parents can’t be reached. Technically and commercially, the developments force trade-offs between personalization/privacy and safety. Regulators may demand detection of minors, logging and reporting of high-risk conversations, and mitigations such as conversation cutoffs, crisis referrals, or mandated escalation — all of which could alter engagement-driven model design and monetization. The FTC probe could expose training, testing and retention practices, while state-level rules risk a patchwork regime that companies have warned against. Ultimately, AI systems built to emulate caring humans may soon be held to higher accountability standards akin to caregivers or regulated services, reshaping product architecture, safety testing, and disclosure practices across the industry.