How do agents see your website? (what-do-agents-see.runtype.app)

A new tool has been introduced that allows developers to inspect how AI agents access their websites compared to human users. Dubbed the Agent Experience (AX) tool, it enables users to fetch a URL twice—once as an AI agent and once as a standard browser—providing side-by-side comparisons of the responses. This is significant for the AI/ML community as many modern websites serve different content to agents, often delivering smaller payloads or even blocking access entirely, which can impede how effectively AI tools can leverage website data. The tool utilizes a Cloudflare Worker to execute parallel fetches, analyzing the responses in terms of body size, status codes, and additional signals relevant to agents, such as robots.txt directives or markdown specifications. With an estimated 97% of sites lacking specific signals for agents, developers can quickly enhance their site’s compatibility with AI tools. Recommendations include utilizing the "Markdown for Agents" feature on Cloudflare to optimize content delivery and adopting the emerging llms.txt specification to direct agents toward crucial content. This initiative underscores the importance of agent visibility in improving interactions between AI models and web content, fostering better data accessibility for future AI applications.