Citations in AI-written reports did not exist (www.stipple.sh)

A recent study evaluated the credibility of citations in AI-generated reports by testing fifteen language models with two different prompts requiring sourced references. The results revealed a concerning fabrication rate, with 38.6% of the 101 citations being completely nonexistent. The research highlighted stark differences between the two topics examined: a widely-covered theme on remote work had a lower fabrication rate of 24.5%, while a more niche discussion on AI-writing detection saw over half (54.2%) of its citations fabricated. These findings carry significant implications for the AI/ML community, as they underscore the need for improved accuracy in AI-generated content. Models can produce the formal structure of citations without substantive backing, demonstrating a reliance on citation shape over genuine source knowledge. This analysis cautions users to verify citations first before accepting claims made within AI reports, particularly in less-documented topics where misinformation may proliferate. As AI continues to be integrated into research and policy-making, ensuring the validity of references will be crucial to maintaining trust in AI-generated outputs.