Anthropic and OpenAI are crawling the web even more and not giving much back (www.businessinsider.com)

Recent data from Cloudflare reveals that AI companies Anthropic and OpenAI are increasingly crawling the web while providing minimal referral traffic to the websites they scrape. The analysis shows a worsening crawl-to-refer ratio, indicating that these AI giants are extracting vast quantities of human-generated data without adequately compensating the original content creators. This shift signifies a troubling departure from the established web model, where content creators received referral traffic in exchange for granting access to their data. The implications of this trend are significant for the AI/ML community and web ecosystem. As companies rely on bots to gather rich datasets for training AI models, they are contributing to higher operational costs for affected website owners, who are left absorbing increased traffic costs without the anticipated benefits of enhanced user engagement. The ethical considerations of this practice are becoming a central debate, as the foundational balance of the web—the mutual exchange of content for traffic—is disrupted. With major players like Google still sending traffic as part of its traditional search model, the mismatch in behavior among AI entities raises questions about the sustainability and fairness of their operations in the evolving digital landscape.