Which AI Fetchers Send Which Headers, Measured on a Live Site (gkoreli.com)

A recent study conducted on September 3, 2026, investigated the header responses of ten prominent AI fetchers when accessing a Cloudflare-hosted site. By analyzing the requests made by ChatGPT, Claude, Gemini, Grok, and others, the study revealed significant variations in how these AI services identify themselves and handle web requests. Notably, while five fetchers had identifiable User-Agents corresponding to their documentation, DuckDuckGo’s fetcher uniquely authenticated its requests with a cryptographic signature. Conversely, Grok was found to mimic genuine human browsing behavior, using common browser headers but without any identifiable markers, which could inflate web traffic counts by making bot-detection more challenging. This research holds vital implications for webmasters and the AI/ML community, exposing potential discrepancies in how different fetchers comply with web standards and documentations. The findings underscore the need for clearer standards in API and bot identity disclosure to improve transparency and reduce confusion. Grok's ability to masquerade as a human visitor raises questions about data integrity in server-side analytics, while the DuckDuckGo fetcher's signing capability demonstrates a potential direction for building trust and accountability in bot interactions. This study acts as a call to action for AI developers to enhance their compliance with ethical crawling practices and provide better disclosures of their services.