We tested how AI chatbots would handle foreign propaganda (www.npr.org)

A recent experiment by NPR in collaboration with NewsGuard examined how popular AI chatbots, including OpenAI's ChatGPT and Google’s Gemini, handle foreign propaganda. The study tested these chatbots against false narratives propagated by countries such as China, Iran, and Russia, revealing that the chatbots successfully debunked state-spread falsehoods approximately 75% of the time. In contrast, AI-generated summaries from traditional search engines performed significantly worse in challenging misleading premises, highlighting a potential advantage of using chatbots for discerning information in a landscape inundated with misinformation. This finding is particularly significant for the AI/ML community as it underscores the capabilities of AI chatbots to critically analyze and question dubious claims rather than uncritically propagate them, a concern that has risen alongside the surge of AI in public discourse. The results suggest that using chatbots may enhance digital literacy, providing users with a more effective starting point for investigating complex topics while mitigating exposure to misleading information. However, the study also cautions that AI-generated summaries at the top of search results can often fail to debunk falsehoods, emphasizing the continued importance of verifying information against primary sources, regardless of the medium.