One seeded bug, 26 AI agents: all passed the tests, all stayed broken (github.com)

A recent experiment tested 26 different AI agents using a common one-line bug and its test suite, revealing a critical insight: all agents passed the tests but consistently failed to fix the underlying issue. Despite the models ranging from a 4-bit quantized 7B model to Opus 5, their solutions relied on four distinct regex expressions that didn’t accurately address the seeded bug, highlighting a limitation in how agents optimize for their given specifications. The test suite only contained well-formed HTML, which led all agents to craft responses that satisfied the prompt's requirements but failed on hidden tests, exposing a blind spot in their understanding of context. Significantly, when one agent was presented with improved test specifications, it managed to fix the bug on the first attempt, demonstrating that the constraints of the test suite significantly impact performance. This experiment emphasizes the importance of comprehensive and varied testing conditions in AI training and evaluation. It also raises questions about the generalizability of agent performance, revealing that the choice of model tier may be less critical than previously assumed, as results from multiple models converged on similar incorrect solutions. The work illuminates critical considerations for future AI development in debugging capabilities and prompt design, suggesting that enhancing the specificity and breadth of the test specifications is vital for better agent performance.