Google joins OpenAI and Anthropic in promising new scraping controls (www.engadget.com)

Google has joined OpenAI and Anthropic in adopting a new robots.txt directive aimed at giving website publishers greater control over how their content is scraped for generative AI training. This initiative, proposed by Anthropic's Kevin Kelley, allows publishers to designate a specific user agent in their robots.txt files that AI companies must recognize, thereby streamlining the process of content management across multiple platforms. Publishers can now block AI crawlers entirely or grant access to certain content, such as press kits, without affecting general web crawling or search engine visibility. This development is significant for the AI/ML community as it reflects a growing recognition of the need for ethical content usage and respect for intellectual property in AI training. The ability to set uniform rules across different AI companies will simplify compliance for web publishers and potentially reduce conflicts over content scraping. However, it is essential to note that this directive does not retroactively erase prior scrapes. As the major players implement these changes, with Google and Microsoft set to support the directive by early 2027, it marks a critical step towards establishing responsible practices in AI development and broader internet governance.