I ran each AI coding-agent sandbox's own tests against the others (github.com)

A new repository has been launched to measure and compare the performance of various AI coding agent sandboxes against standardized tests. This initiative addresses the often-unsubstantiated claims made by sandbox creators regarding their agents' security features, such as restrictions on accessing sensitive data or external networks. The repository allows users to run specific contracts—essentially tests that verify whether a sandbox meets promised security and functionality standards—and publish the results, fostering transparency and accountability in the AI/ML community. This development is significant as it provides a systematic approach to evaluate the security of AI coding environments, which are crucial for protecting sensitive information. The framework categorizes tests into three parts: ‘attack’ (identifying what the agent shouldn't do), ‘control’ (ensuring core functions remain operational), and ‘effect’ (verifying that no unauthorized changes occurred post-execution). With an initial set of 196 contracts sourced from various trusted public test suites, including sandbox-runtime and Codex, the repository establishes a clear baseline for future evaluations. Those involved in AI development can now utilize these insights to enhance the security measures of their coding environments, contributing to a more robust AI landscape.