Llms.txt: agent-optimization wishful thinking (www.phpied.com)

A recent exploration into the functionality of an "llms.txt" file—a proposed counterpart to the traditional robots.txt—has sparked debate in the AI/ML community regarding its viability for optimizing agent interactions with web content. The creator of sightread.org initially envisioned the llms.txt as a resource where LLMs could find structured URLs for exercises, only to encounter skepticism from various AI models like Gemini and Claude, which suggested that crawlers largely ignore the file and that a public HTML guide would be more effective. Despite attempts to format llms.txt appropriately and integrate it with standard web practices, results from server logs indicated minimal crawler requests, raising doubts about its overall significance. The significance of this inquiry lies in the broader implications for how LLMs interact with web resources and the methods developers use to enhance AI capabilities. The stark contrast between crawler interactions with llms.txt and traditional resources like robots.txt highlights a potential gap in utilizing LLMs effectively in web navigation. While Lighthouse has introduced audits to evaluate agent browsing, the observed low engagement from major AI-based crawlers points to a critical need for developers to reassess the utility of such files, ultimately suggesting that conventional methods may still reign supreme in guiding AI traffic.