No one is evaluating AI coding agents in the way they are used (marginlab.ai)

A recent analysis reveals significant gaps in how AI coding agents are evaluated compared to their real-world usage, highlighting discrepancies between scores reported by frontier labs and official benchmark platforms like SWE-Bench-Pro. This discrepancy arises because coding agents are often assessed in ways that do not reflect their practical application. For instance, while labs optimize their evaluations using advanced scaffolding, official benchmarks tend to rely on less sophisticated setups and static scoring that may fail to capture the dynamic nature of model updates and performance in operational settings. The implications for the AI/ML community are critical, as traditional benchmarks may misrepresent the true capabilities of these models, leading developers to make unintentionally misguided choices about tool usage. To address this, MarginLab is pioneering evaluations that align with the actual conditions under which these models are employed, regularly updating evaluations to reflect new scaffold features and configurations. This innovative approach aims to provide more accurate assessments, empowering developers to make informed decisions on which model-scaffold combinations yield the best performance at reasonable costs.