Inside the behind the scenes battle for the future of the web (www.businessinsider.com)

A critical battle is unfolding within the Internet Engineering Task Force (IETF) over new web standards that could drastically reshape how AI systems access online content. Traditionally, search engines like Google and Bing crawled and indexed websites to direct users to original sources, ensuring creators received traffic that supported their revenue models. However, AI-powered answer engines—such as Google’s AI Overviews and OpenAI’s ChatGPT—deliver direct responses by scraping web content without sending users back to the source, threatening the economic foundation that funds high-quality online content creation. The IETF is drafting standards to differentiate traditional search engines from generative AI systems, enabling website owners to selectively block AI bots from scraping their data for model training or outputs while still allowing conventional search bots that drive referral traffic. This distinction centers on the principle that search engines should send users to the original content, a criterion AI answer engines often fail to meet. Big Tech companies including Google, Microsoft, and OpenAI oppose this split, arguing that AI and search are inseparable and warning that the definitions could disrupt search results or invite regulatory scrutiny. Meanwhile, publishers push for greater control to protect their content and revenues from AI-driven exploitation. These standards, likely finalized by 2025, could redefine web crawling norms and influence regulatory approaches to AI data use. They bear significant implications for AI development, balancing the need for high-quality training data against creators’ rights and financial sustainability. This ongoing debate reflects a broader tension between innovation in AI-powered information delivery and preserving the web’s economic ecosystem.