We need to start doing web blocking for non-technical reasons (utcc.utoronto.ca)

Blog owner Chris Siebenmann says he’s begun blocking requests that look like “old Chrome” browser user‑agents because a surge of high‑volume crawlers — apparently gathering data for LLM training — are overloading his sites. His anti‑crawler rules treat many outdated UA strings as suspicious, so legitimate visitors using older browsers or archival proxies (archive.today, archive.ph, etc.) can be mistakenly blocked. He calls out archive.* services for using old Chrome UAs, widely distributed IP ranges and even falsified reverse DNS entries that mimic googlebot, and recommends archive.org as a better‑behaved archival crawler. If you’re blocked in error he asks you to email him with your exact User‑Agent string. For the AI/ML community this is a clear signal that site owners are starting to take non‑technical (ethical, consent and capacity) measures to limit large‑scale scraping for model training. Technically, naive defenses based on UA strings, IP ranges or reverse‑DNS lookups are brittle and cause false positives; conversely, widespread UA spoofing and distributed crawling make reliable identification hard. The post highlights practical implications for dataset builders (rate limits, missing sources, legal/ethical exposure) and argues for better norms and tooling: transparent crawler identification, authenticated APIs or explicit opt‑ins/opt‑outs, and respect for site capacity and consent to avoid fracturing the web ecosystem researchers rely on.