The AI-Scraping Free-for-All Is Coming to an End (nymag.com)

After years of mostly unchecked web scraping to train large language models — from experimental crawls to commercialized, often controversial data grabs — a coordinated push is now underway to push back. Cloudflare earlier unveiled tools to track AI scraping and prototype a marketplace to let sites charge for ingestion, and this week a coalition including Reddit, Medium, Quora and Fastly announced the RSL (Really Simply Licensing) standard. RSL aims to let publishers declare whether content can be scraped, how it must be attributed, and what price (if any) must be paid — essentially a machine-readable, monetizable equivalent of robots.txt or RSS for AI ingestion. The technical and industry implications are far-reaching. By combining licensing metadata with enforcement at the CDN/infrastructure layer (blocking, fingerprinting, tracing crawlers), websites could default to being invisible to many AI crawlers, degrading models’ access to fresh news, research, and culture unless firms negotiate licenses or build alternative data pipelines. Enforcement will still be an arms race — scrapers already masquerade as users or search engines and use distributed crawls — and major AI firms may resist paying or find workarounds. But with infrastructure providers now aligned with publishers, the economics of training data could shift from near‑free mass scraping to negotiated licensing, forcing changes in dataset composition, model freshness, and the downstream competitiveness of AI products.