Creative Commons announces tentative support for AI ‘pay-to-crawl’ systems (techcrunch.com)

Creative Commons (CC) has tentatively expressed support for “pay-to-crawl” systems, which propose a method for compensating website owners when their content is accessed by AI webcrawlers. This endorsement follows CC's earlier announcement of a framework for fostering an open AI ecosystem, aimed at facilitating dataset sharing between content owners and AI developers. Under the pay-to-crawl model, companies, such as Cloudflare, would charge AI bots for scraping website content, a shift from the previous model that benefitted publishers through increased traffic from search engines. The proposed system could provide a revenue stream for publishers adversely affected by AI-generated responses that limit user clicks on original sources. While CC sees potential in pay-to-crawl systems for supporting content creators—especially smaller publishers who lack negotiating power—there are concerns about its implications for web accessibility. CC warns that such systems could centralize power and block access for essential public interest users like researchers and educators. They advocate for responsible implementation, suggesting principles that include preserving public access, allowing throttling rather than outright blocking, and ensuring open and interoperable technologies. This initiative reflects a broader trend among tech companies, including Microsoft, towards creating fairer systems for the use of web content in AI training.