From Public Posts to AI-Search Citations: Measuring the Fragility of AI Search (arxiv.org)

A new study highlights the vulnerabilities inherent in AI search systems, revealing how the process of citation selection can significantly influence the integrity and reliability of information retrieved. Unlike traditional search engines that rank pages based on keywords, AI search platforms layer citation processes that can amplify biases by favoring certain sources over others. This raises critical security concerns, particularly as the ease of publication on various platforms can unintentionally lead to the inclusion of low-quality or deceptive information in AI-generated responses. Using a comprehensive measurement framework, researchers analyzed over 17,000 citations across 10 AI search platforms. Findings indicate that citations often concentrate on a narrow set of sources, with top domains capturing up to 70% of citations, many of which have low publication barriers. Experimentally, the study confirmed that new content could quickly influence AI outputs—within just days, a fabricated concept was cited by 8 out of 10 platforms after being posted on easily accessible sites. These insights underscore the urgent need for transparency in citation practices within AI systems to preserve information quality and combat misinformation effectively.