Benchmarking Opus 5 on SlopCodeBench (github.com)

A recent evaluation of Opus 5 using SlopCodeBench, a long-horizon coding benchmark from UW Madison, shows significant advancements yet persistent challenges in AI-driven code generation. SlopCodeBench differs from traditional benchmarks by revealing problem requirements incrementally, requiring models to adapt their codebases over time. In initial tests, Opus 5 achieved a 24% strict pass rate across challenges, only slightly outperforming Opus 4.6's 17%. Notably, all tested models exhibited increased verbosity and code complexity, with Opus 5 generating five times the number of functions as its predecessor. The significance of this benchmark lies in its potential to reflect real-world coding practices, where software evolves through iterative problem-solving. None of the models successfully completed the challenges without accruing defects, highlighting that current AI models still require oversight in software engineering tasks. The results suggest a growing understanding of AI's limitations in producing maintainable code, especially as models struggle with complexity and defects over time. Key metrics from the tests indicate trends in code quality, such as function duplication and complexity, raising questions about the efficacy of AI-generated solutions in maintaining high standards in software development.