Crawlable – We audited 50 top web apps for AI crawler readability (usecrawlable.com)

Crawlable recently released the 2026 AI Search Readability Index, auditing 50 top tech, startup, and e-commerce websites to assess their AI crawler readability. The findings were concerning: none of the sites scored an A, with an average readability score of only 51.5 out of 100. Notably, 78% of the sites failed to name any AI crawlers in their robots.txt files, while 70% did not maintain a compliant llms.txt, which governs how AI models access their content. Additionally, 30% of the sites displayed a common issue of serving empty JavaScript shells, causing delayed text rendering after page loads. This report is significant for the AI/ML community as it highlights a widespread lack of optimization for AI crawlers among leading websites, potentially impacting how AI systems interact with online content. The absence of well-structured llms.txt files compromises proper indexing and retrieval of content by AI models, limiting their effectiveness. These results underscore the need for better adherence to web standards, which could enhance AI-driven search engines' ability to access and process information efficiently. As businesses strive to improve their AI interfaces, addressing these shortcomings could lead to more accessible and responsive online experiences.