MindEval: New benchmark shows 12 top LLMs fail at mental health care (swordhealth.com)

Sword Health has announced the launch of MindEval, a novel benchmarking framework designed to rigorously evaluate Large Language Models (LLMs) in the context of mental health care. With over one billion people worldwide experiencing mental health conditions, the need for effective scalable solutions is critical. MindEval was created in collaboration with licensed Clinical Psychologists to automate the assessment of LLMs, focusing on their ability to engage in realistic, multi-turn therapeutic conversations. By emphasizing clinical competence over mere knowledge, MindEval aims to provide a robust standard for developers and researchers to measure the therapeutic capabilities of AI models. The significance of MindEval lies in its potential to enhance the safety and efficacy of AI in mental health settings, a field currently facing scrutiny for the limitations of existing models. Initial benchmarks indicated that 12 leading LLMs, including GPT-5 and Claude 4.5, fell short of clinical reliability, scoring below 4 out of 6 on average. Key weaknesses emerged, especially in scenarios involving severe symptoms and prolonged interactions, highlighting that larger model sizes do not necessarily equate to superior therapeutic support. By open-sourcing the MindEval framework, Sword Health aspires to foster collaboration and innovation in building AI systems that can reliably meet the nuanced demands of mental health care.