Web­Space Invaders (matthiasott.com)

A recent incident has highlighted a growing challenge in the AI and web hosting sectors: a surge in bot traffic primarily driven by large language models (LLMs) from major companies, such as OpenAI and Meta. The author discovered that their personal website was being bombarded with access requests from LLM crawlers, leading their hosting provider to implement drastic measures, blocking access from certain regions in an attempt to mitigate what they misidentified as a DDoS attack. This situation underscores the ongoing struggle faced by small website owners, who must contend with a disproportionate volume of traffic from automated agents while trying to maintain their sites for human users. The significance of this issue extends beyond personal grievances to encompass broader concerns within the AI/ML community and the ecosystem of the open web. As bots account for nearly 50% of all web traffic currently, the relentless scraping of content for training AI models threatens the viability of small sites. The power dynamics are increasingly skewed, with independent content creators shouldering the burden of increased operational costs while tech giants benefit from harvesting their data. This raises critical questions about the sustainability of the open web and the extent to which websites should be reliant on third-party solutions, like Cloudflare, for protection against the evolving landscape of intelligent, adaptable scraping bots.