Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping (www.techradar.com)

Google, Microsoft, OpenAI, and Anthropic have announced a shared robots.txt standard aimed at streamlining how websites can opt out of having their content scraped for AI training. This initiative allows webmasters to use a single token, "__usage_ai-training", to express their preferences regarding the use of their content, significantly reducing the complexity of managing rules for multiple crawlers. However, it’s important to note that this standard does not prevent bots from accessing the content; instead, it merely guides how that content may be used for generative AI training, leaving existing scraped data and current crawling practices unchanged. This collaborative effort signifies a notable step in addressing website owners' concerns about data scraping, though it raises questions about effectiveness, as the commitment to adhere to the new standard is voluntary and varies among companies. While Anthropic has begun implementing the standard, OpenAI and Microsoft are scheduled to follow suit in the upcoming years, with Google also targeting 2027. The announcement, while positive, emphasizes that these changes may not drastically alter the landscape of AI content usage, and existing control measures for web scraping will remain necessary.