Google, Anthropic and others to support new AI crawling opt-out standard (siliconangle.com)

Google, Microsoft, Anthropic, and OpenAI have announced their support for a new website crawling standard aimed at allowing content publishers to easily control how AI developers utilize their data for model training. This initiative seeks to enhance the existing robots.txt file system, enabling website operators to block all training-related scraping from generative AI models with a single line, rather than requiring a separate entry for each AI provider. This change addresses the growing concerns over data usage and copyright in the AI field, allowing websites to more effectively safeguard their content. The significance of this development lies in its potential to streamline compliance for web publishers while promoting ethical data usage by AI companies. Following the standardized Robots Exclusion Protocol, the proposed modifications are in the draft stage and will be submitted to the Internet Engineering Task Force for approval. In addition to this new standard, Microsoft plans to introduce an "AI Grounding" feature, enabling operators to prevent AI models from accessing their sites for non-training purposes, with similar features being implemented by OpenAI. This collective effort highlights the AI/ML community's recognition of proper data sourcing and the evolving dynamics of content ownership in the digital age.