The biggest names in AI are backing a new way to opt out of scraping (www.theverge.com)

OpenAI, Google, Microsoft, and Anthropic have announced their commitment to a new standard that allows publishers to opt out of having their content scraped by AI systems. This initiative, proposed by Kevin Kelley from Anthropic and backed by major players in the AI space, introduces a shared "policy token" that publishers can include in their robots.txt file. This token will streamline the process, enabling publishers to set their AI training preferences across multiple companies rather than managing each individually, thereby enhancing their control over how their content is utilized by AI. This development is significant for the AI/ML community as it addresses the growing concerns of publishers regarding unauthorized data scraping as AI technologies proliferate. By adopting this shared standard, AI companies are recognizing the importance of respecting publishers' rights and preferences, especially in an era where data is a critical component of AI training. Microsoft has indicated that it will provide granular controls for publishers, allowing them to opt out of AI training while still maintaining visibility in search results. With Anthropic already implementing this sign, Google and Microsoft are expected to follow suit next year, marking a pivotal shift in how AI companies interact with content creators and laying the groundwork for industry-wide compliance and coordination.