Harness overhead benchmark: tokens 11 coding agents send before the task starts (testdino.com)

A recent benchmark study evaluated the token overhead generated by 11 coding agents before executing a task, revealing that agents sent between 1,894 and 31,798 tokens merely to respond with the word "OK." This significant insight underscores that a considerable amount of data is processed behind the scenes before any meaningful computation occurs, with agents like Kiro CLI and Claude Code leading in overhead due to extensive tool schemas. Understanding this token overhead is crucial for the AI/ML community, as it highlights the efficiency of different harness models and offers a standardized metric for future comparisons. The study emphasized the importance of isolating the harness from the model's performance. By comparing agents based solely on their pre-execution token counts, it allows for a clearer assessment of their efficiency, without the complexities of task variability. This benchmarking method can guide developers in selecting effective coding agents, especially in resource-constrained environments. Moreover, the implications extend to long-term usage, as the first call's overhead compounds with subsequent requests, potentially resulting in millions of tokens used during full task execution.