The Benchmark Gap: 1,472 runs show coding-agent context changes outcomes (github.com)

A recent independent study by Doruk Ardahan analyzed 1,472 runs of the GLM-5.1 model in OpenCode, highlighting a significant discrepancy between vendor-reported benchmarks and real-world tool performance. The research found that while Z.AI reported a 75.9% success rate at 32K context with GLM-5, the actual performance in a typical coding environment dropped to 0%. The study indicated that substantial built-in context overhead—approximately 21K tokens—at this setting severely limited effective working context, leading to heightened failure rates. In contrast, the model performed exceptionally well at 80K context, achieving 100% success in preserved thinking conditions. This exploration is noteworthy for the AI and machine learning community as it underscores the importance of evaluating AI coding tools in realistic environments rather than relying solely on vendor benchmarks that may employ optimized setups. The findings suggest that tool builders should account for inherent background context in their models since it can dramatically influence performance outcomes. Researchers are urged to specify runtime environments in published benchmarks to ensure accurate comparisons. This study ultimately demonstrates the necessity of transparency in model evaluation and the potential pitfalls of relying on theoretical performance metrics.