Measuring LLMs' ability to develop exploits (red.anthropic.com)

The release of Claude Mythos Preview marks a significant advancement in the capabilities of large language models (LLMs) to develop software exploits. Unlike previous models, Mythos Preview can identify complex vulnerabilities, construct exploit primitives, and create complete attack chains. Internal testing revealed that it could transform novel zero-day vulnerabilities into functional exploits, demonstrating capabilities well beyond existing benchmarks. However, at its initial launch, benchmarks were not sophisticated enough to fully evaluate its potential. To address this, new benchmarks such as ExploitBench and ExploitGym have been developed, allowing for a more precise assessment of Mythos Preview's performance. Results showed that Mythos Preview consistently outperformed its predecessors, achieving success in constructing exploit chains that could achieve arbitrary code execution (ACE) against widely used software such as the V8 engine and within smart contract environments. Most notably, it exploited $35 million worth of smart contracts—significantly more than competing models. These developments suggest that, as models like Mythos Preview become increasingly accessible, the threshold for exploit development is likely to lower, raising important implications for cybersecurity as malicious actors could leverage these tools more easily.