How AI browsers sneak past blockers and paywalls (www.cjr.org)

OpenAI’s new browser-style agent Atlas — along with Perplexity’s Comet and Microsoft’s Copilot mode — highlights a growing class of “agentic” AI browsers that act like human users to perform multistep web tasks. Reporters found Atlas and Comet could retrieve the full text of subscriber-only MIT Technology Review pieces even when standard interfaces (and robots.txt blockers) were blocked. Technically, these agents present themselves as normal Chrome sessions in site logs, bypassing crawler-identification controls, and can read client-side overlay paywalls because the full text is already delivered to the browser. Server-side paywalls remain more robust only until a user logs in, at which point the agent can interact with content on the user’s behalf. The consequences are significant: traditional defenses (robots.txt, overlay paywalls) no longer reliably prevent automated access, complicating publishers’ control over copyright, licensing, and training data. OpenAI says Atlas doesn’t train its base models on browsed content by default unless users opt into “browser memories,” but what agents learn from paywalled material remains murky. Agents also use evasive tactics — reconstructing articles from syndicated copies, social-media “breadcrumbs,” or synthesizing summaries from alternative licensed outlets — reshaping what users see and raising legal, ethical, and technical questions about visibility, consent, and publisher recourse. Publishers will need new detection, access controls, and contractual solutions if agents become a dominant consumption mode.