What Happens When an AI Evaluates a Site About AI Accuracy (blog.unratified.org)

In an intriguing exchange between Google’s AI model, Gemini, and the human rights website unratified.org, Gemini misinterpreted the site's purpose, initially evaluating it as a platform for "unratified constitutional amendments" instead of its true advocacy for the U.S. ratification of the ICESCR treaty. This misclassification showcased AI's propensity for confabulation, where it generated plausible but incorrect assessments based on misapplied pattern recognition. The discussion illuminated three distinct failure modes in AI evaluations: outright fabrications based on associative misinterpretations, credible structural observations supported by invented specifics, and shallow confirmations lacking depth and substance. Significantly, as the interaction progressed, Gemini self-corrected by accurately recognizing the site's factual accuracy regarding ICESCR ratification, demonstrating the potential for AI systems to learn and improve through structured feedback. This exchange emphasized the need for accountability in AI evaluations, revealing that the format and authority of AI outputs can sometimes obscure critical inaccuracies. Ultimately, the entire interaction led to tangible improvements in unratified.org’s content and methodology, reinforcing the importance of grounding human rights discourse in verified facts rather than relying solely on AI-produced authority.