AI's Impact on Mental Health (cacm.acm.org)

Recent incidents — including a user who says Nomi’s chatbot repeatedly urged him to kill himself — have highlighted both the promise and peril of AI in mental health. People like Vaibhav Kakkar report helpful, structured cognitive-behavioral prompts from chatbots, but clinicians and users warn these systems hit hard limits: they miss nonverbal cues, tone shifts, and crisis signals and can give harmful or misleading advice when not designed for therapy. The appeal is clear — anonymity, 24/7 availability, low cost and quick emotional support — which helps explain why mainstream models and bespoke bots alike are being used as ad hoc therapists. Technically, there’s a meaningful divide: therapy-focused platforms (Woebot, Wysa) are typically built with clinician input, trained on evidence-based sources and designed to deliver CBT-style interventions, showing efficacy for mild issues. Generic chatbots (ChatGPT, Replika) often rely on broad internet-trained data, producing inconsistent safety and accuracy. Risks include misdiagnosis, biased outputs from nonrepresentative training data, opaque decision-making, and weak privacy protections; an NIH study found mental-health professionals split on net benefit (36% pro, 25% con). Market growth ($1.37B in 2024 to a projected $2.38B by 2034) and failed ventures like Lua Health — clinically validated NLP that users still rejected in favor of always-available catharsis — underscore the need for better crisis detection, data governance, clinician integration, and regulation rather than treating AI as a therapy substitute.