Cloudflare CEO Matthew Prince is pushing UK regulator to unbundle Google’s search and AI crawlers (techcrunch.com)

Cloudflare CEO Matthew Prince told the U.K. Competition and Markets Authority (CMA) that Google’s practice of using the same web crawler for both search and its AI products gives it an unfair competitive edge, and urged regulators to “unbundle” those crawlers. Cloudflare — which recently launched a marketplace enabling sites to charge AI bots for scraping content and counts roughly 80% of AI companies among its customers — provided the CMA with data on how Google’s crawler operates. The CMA has already granted Google a special status in search and ads, which allows it to consider broader interventions affecting Google’s AI Overviews, AI Mode, Discover feed, Top Stories and News tab. Technically, Prince argues Google’s bundled crawler means sites can’t opt Google’s AI scraping out without also impairing search-related services (including ad safety), forcing publishers to permit free access that rivals (Anthropic, OpenAI, Perplexity) would otherwise have to pay for. That bundling affects training-data access, cost structures and market entry for LLM providers; blocking Google can disrupt ad delivery and publisher revenue, so many publishers feel forced to allow crawling. If regulators require separation or if pay-for-content markets expand, it could reshape how training datasets are acquired and priced, create new licensing dynamics, and open space for thousands of AI firms to compete to buy content rather than rely on unfettered crawling. Publishers already using Cloudflare’s controls say deal talks with major LLMs are underway.