Study: AI responses to healthcare queries are nearly 76% accurate (www.psu.edu)

A recent study from Penn State University has revealed that AI-powered chatbots achieve approximately 76% accuracy in answering everyday health-related questions, highlighting significant concerns about their reliability for patient care. The research team aimed to assess how the general public utilizes AI, particularly large language models (LLMs) like ChatGPT, for healthcare queries and the potential risks associated with inaccurate information. They discovered that while these tools can be useful, they are best utilized by trained healthcare professionals rather than lay users, especially in specialized fields such as neurology and dermatology. To conduct their analysis, researchers organized a "Diagnose-a-thon" competition, where 34 participants employed four different LLMs to respond to real and hypothetical health inquiries. Medical evaluations by nine board-certified physicians assessed the accuracy and potential harm of the responses. The significance of these findings lies in their reflection of real-world usage patterns and the necessity for caution when seeking medical advice from AI. This study not only emphasizes the need for further exploration of AI's role in healthcare but also calls for the development of more robust algorithms that can minimize risks associated with misinformation in patient care settings.