Free and open source software is incompatible with (security) guarantees (utcc.utoronto.ca)

Author Chris Siebenmann posted that he’s blocking suspicious browser versions from accessing his blog and wiki because a surge of high-volume crawlers — many using old Chrome User-Agent strings — appear to be harvesting content, reportedly in part for LLM training. His anti-crawler heuristics treat these outdated UAs as suspicious, which can accidentally block legitimate visitors; he asks anyone blocked to email him (with their exact User-Agent) so he can investigate. He also warns that some archival services (archive.today/phi variants) crawl in ways indistinguishable from malicious actors: they use old Chrome UAs, operate from widely distributed IP ranges, and some even present falsified reverse DNS records claiming to be googlebot. He recommends archive.org as a better-behaved alternative. Technically, the post highlights practical detection and mitigation tactics — relying on User-Agent strings, IP-blocking, and reverse-DNS checks — and exposes their limits. For the AI/ML community this is significant: indiscriminate scraping for model training is driving site operators to deploy blunt defenses that interfere with benign crawlers and web archiving, complicate lawful data collection, and reduce reproducibility. The note underscores a growing tension between large-scale data harvesting and site operators’ need to protect server load and integrity, pointing to a need for clearer crawler identification standards, consent mechanisms, and responsible data practices.