Show HN: Which big sites block AI training crawlers but let AI search in (getkavel.com)

A recent analysis of 945 prominent websites revealed that many block AI training crawlers while permitting access to AI search crawlers. Conducted in October 2026, the study found that 193 of the websites prevented at least one AI crawler in their robots.txt files, with 69 sites specifically blocking AI training but allowing AI search activities. Notably, popular websites like Airbnb, BBCGoodFood, and Netflix were identified in this category. This behavior highlights a growing trend among major platforms to safeguard their content against data scraping for training AI models, possibly to maintain control over their intellectual property. The findings underscore a significant shift in how companies regulate AI access to their data, raising implications for AI development, data collection strategies, and compliance with regulatory environment shifts. Sites primarily focused on news and reference material were the most likely to block training crawlers, suggesting a protective stance against the unauthorized use of their data. In contrast, the opening of search crawlers illustrates a complex relationship where these companies seek to benefit from AI-driven search capabilities, emphasizing the ongoing dialogue in the tech community regarding data usage rights amid the evolving landscape of AI technologies.