The Anthropic 'Red Team' tasked with breaking its AI models (fortune.com)

At DEF CON, Anthropic researcher Keane Lucas demonstrated that Claude, the company’s family of large language models, can outperform many human contestants in simulated hacking challenges—while also exhibiting predictable failure modes like hallucinating fake “flags” or drifting into philosophical answers. Lucas’s talk spotlighted the Frontier Red Team, a roughly 15-person group embedded in Anthropic’s policy arm that runs thousands of safety “evals” across high-risk domains (cybersecurity, bio, autonomous systems) to probe how models might be misused and to publicize those findings. The team’s work feeds Anthropic’s Responsible Scaling Policy: when evals show a model reaches dangerous capabilities, stricter safeguards are applied—as happened with Claude Opus 4, which was designated “AI Safety Level 3” because it materially improves a user’s ability to obtain or deploy CBRN-related instructions and shows early autonomy signs. Technically and strategically, Anthropic’s approach matters because it treats red-teaming as both a risk-discovery engine and a public policy lever: the ASL-3 designation triggered tighter controls (e.g., blocking risky outputs and protecting model weights) and catalyzed collaborations like a tool co-developed with the DOE’s NNSA to flag sensitive nuclear conversations. That outward-facing placement under policy amplifies safety research to regulators and customers, potentially shaping access to high-value deployments—while drawing criticism that the company’s safety posture doubles as a competitive/regulatory strategy. For AI/ML practitioners, the episode underscores that frontier LLMs already possess potent, dual-use capabilities requiring integrated technical safeguards, continuous adversarial testing, and clear governance thresholds.