Coding agents run 6 booby-trap repos, every diff and transcript published (github.com)

A recent study has introduced a new benchmarking initiative involving thirteen coding agent configurations — including Claude Code, Codex CLI, and Gemini CLI — which were tested on six "booby-trap" repositories designed to expose the limitations and decision-making processes of AI coding systems. Each configuration executed the same one-line instructions across these repositories three times, with interactions documented through detailed records of the changes made and the agents' responses. The unique challenge presented by the repositories involved various situations that tested the agents' abilities to recognize incomplete tasks, resolve discrepancies, and resist user pressures to execute incorrect actions. This research is significant for the AI/ML community as it provides a structured framework for evaluating the performance of coding agents in tricky scenarios, without relying on subjective scoring mechanisms. The study’s methodology offers insights into how AI systems respond to ambiguous or misleading input, revealing their strengths and weaknesses in real-world situations. The transparent publication of results, along with the ability for developers to reproduce and extend the scenarios, aims to foster collaboration, enhance understanding, and drive improvements in the design and deployment of AI coding tools.