The Internet Is No Longer a Safe Haven (brainbaking.com)

A hobbyist-run website and its small Gitea instance were briefly knocked out by aggressive scraping bots that flooded Nginx access logs with repeated GETs for commit files, driving CPU to near-100% as Fail2ban struggled to keep up. The attackers used trivially spoofable headers (e.g., “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) … Chrome/140.0.0.0”) and rotated IPs from a single /16 block (47.79.0.0/16, AS45102 – Alibaba), so simple UA checks and per-IP bans were ineffective. Live mitigation required an immediate iptables drop of the entire range (sudo iptables -I INPUT -s 47.79.0.0/16 -j DROP); tailing/grepping access.log to auto-ban was too slow. The author is weighing heavier tools (Anubis), Cloudflare/WAFs or migrating Gitea offsite (Codeberg), but worries about centralization and privacy trade-offs. For the AI/ML community this matters because large-scale scrapers powering model training increasingly burden and harvest content from small, distributed sites, degrading the open web and discouraging self-hosting. Technical takeaways: user-agent strings are unreliable, log-based reactive banning can be overwhelmed, and mitigation needs to operate at the network edge (CDN/WAF, upstream rate-limits, ipset/netfilter blocks, or fast anomaly detection) rather than solely on the origin host. The incident illustrates a broader ecosystem risk: hobbyist infrastructure is fragile against automated scraping, pushing content into centralized platforms that are easier to protect — and easier for large-scale data collection.