You Don't Need Anubis (fxgn.dev)

AI training scrapers have become increasingly aggressive—ignoring robots.txt, spoofing user agents and IPs, and hammering small sites—so many site owners adopted Anubis, a proof-of-work (PoW) JavaScript gate that forces visitors to solve a tiny cryptographic puzzle before loading content. The author argues Anubis is useful for real DDoS protection but overkill for the common problem (aggressive LLM scrapers like ClaudeBot) because most scraping bots simply don’t execute JavaScript. They demonstrate a far lighter approach: a 12-line Caddyfile that serves a tiny JS page to visitors without a “verified” cookie, sets the cookie, reloads, and proxies normal traffic—effectively blocking non-JS bots with minimal user friction and near-zero latency compared to Anubis’s slower PoW page. For the AI/ML community this highlights two things: first, the current scraping arms race is often won by simple client-side heuristics, not heavy cryptography, because scraping costs to large model trainers are still negligible and many bots avoid JS. Second, it underscores a temporary, brittle equilibrium—bots may start running JS, at which point both cookie-gates and Anubis can be bypassed. For genuine DDoS or long-term protection, centralized services like Cloudflare remain more robust, but for the common anti-scraping use case, lightweight invisible defenses are a practical, less user-hostile alternative.