I crawled 1,500 sites: 30% block AI bots, 0.2% use llms.txt (websiteaiscore.com)

A recent analysis of 1,500 websites highlights significant gaps in web readiness for AI-driven search technologies, revealing that a staggering 30% of websites inadvertently block AI crawlers due to outdated robots.txt configurations. This unintended barrier can severely hinder a site's visibility in the burgeoning AI economy, as blocking bots like GPTBot prevents crucial citations from being obtained. The study suggests that sites adopt a strategic robots.txt policy to differentiate between "Search Bots" and "Training Bots" to optimize their accessibility. Moreover, the audit underscored a pronounced "Semantic Void," with 70% of sites lacking structured data, crucial for AI understanding. Only a minuscule 0.2% utilized the new llms.txt file, which could markedly improve AI comprehension of valuable content by reducing unnecessary crawl time and computational costs. Other key findings pointed to inefficiencies such as high token budgets due to bloated HTML and significant reliance on JavaScript, which can confuse real-time AI agents. As the research indicates, websites that address these structural shortcomings will be better positioned to leverage AI technologies, underscoring a shift towards prioritizing web structures over mere visual appeal in the era of Answer Engine Optimization (AEO).