Is llms.txt is of any use? (www.soniclinker.com)

Recent analysis of AI-agent traffic by SonicLinker revealed that none of the monitored requests—totaling approximately 2 million monthly—were made for the proposed standard file /llms.txt. This file is intended to help large language models (LLMs) effectively navigate website content, similar to how robots.txt assists search engines. Despite its promise for aiding AI agents in retrieving essential information from sites, the findings indicate that popular AI platforms like ChatGPT and Claude are not utilizing this standard, opting instead to directly fetch and parse normal web pages. The significance of this observation lies in highlighting the existing behavior of AI agents, which currently rely on established web navigation methods rather than a specialized discovery file like /llms.txt. Findings suggest that modern LLMs excel at extracting meaningful content from well-structured HTML and straightforward documentation, making the urgency to implement llms.txt questionable for now. As AI continues to shape web consumption, there may be a paradigm shift toward developing inherently AI-readable websites, focusing on simplifying content structures to enhance LLM accessibility without the need for additional metadata files.