When AI Health Advice Fails, the Failure Is Not Accuracy. It Is Evidence (www.aivojournal.org)

Recent reports highlight a significant governance failure in Google’s AI Overviews, which provided misleading health advice without a traceable basis. The incidents, including inaccurate dietary recommendations for pancreatic cancer patients and misinterpretations of liver test results, raised concerns about the lack of reconstructable evidence to verify the advice given. Upon challenge, neither users nor the platform could produce evidence of what was shown, undermining the integrity of the AI’s health-related outputs. This situation underscores a critical shift in how the AI/ML community must approach governance in AI systems, especially when they deliver health information. Current oversight frameworks, such as the AIVO Standard, advocate for the mandatory capture of "Reasoning Claim Tokens" (RCTs)—evidence artifacts that document the specifics of AI-generated responses. Without these records, accountability and oversight become problematic, as disputes cannot be meaningfully resolved due to the absence of contextual evidence. As AI systems increasingly serve as authoritative sources of information in regulated domains, the necessity for robust evidentiary controls becomes clear, marking a crucial evolution in the governance of AI in healthcare and beyond.