The Code They Leave Behind: Beyond Benchmark Scores (bitbin.de)

A new discussion in AI development is challenging the conventional metrics used in coding benchmarks, particularly the binary pass/fail approach. The release of Opus 5, which scored impressively on Terminal Bench 4.0, illustrates this gap—developers have found it to be practically unusable despite its high benchmark score. This discrepancy raises critical questions about what is truly being measured in AI models, emphasizing the need for deeper analyses that go beyond numerical scores to assess the qualitative aspects of code produced. To address this, five new programmatic views of model performance are proposed: reliability (consistency in task completion), verbosity (amount of code generated), complexity (fragility and structure of code), specialization (performance variability across domains), and writing quality (clarity and reasoning in code). These criteria offer a more nuanced understanding of AI's contributions to coding—highlighting issues such as Opus 5's complexity and verbosity compared to other models. By incorporating subjective experiences and static analysis, the AI/ML community can begin to move towards a more comprehensive evaluation of coding performance, paving the way for improvements that better align model capabilities with real-world usability.