AI bots ignore evidence. Can we trust them with science? (www.sciencenews.org)

Recent experiments revealed significant limitations in AI chatbots like ChatGPT and Gemini when it comes to scientific reasoning. YouTuber FatherPhi demonstrated that these models struggle to update their predictions based on new evidence, as seen when they incorrectly predicted the behavior of a pen under simple physical principles. This tendency persists across various tasks, suggesting that AI lacks the iterative reasoning process that human scientists employ. A study showing AI agents ignoring evidence during scientific reasoning tasks highlighted that these systems made unsupported claims in 53% of tests and only properly incorporated contradictory evidence 26% of the time. This finding is crucial for the AI/ML community, raising questions about the reliability of AI in scientific contexts where evidence-based decision-making is essential. While researchers are exploring “reasoning models” designed to enhance the reasoning capabilities of AI, doubts remain that these models genuinely think through problems. Experts caution that while AI can assist in well-defined tasks, it is not yet ready for complex scientific inquiries. As the narrative around AI’s intelligence evolves, there is a growing call within the community to critically assess and improve these systems to unlock their potential for meaningful scientific advancements.