🤖 AI Summary
A recent study conducted by Baseten and announced the launch of the v1.1 FrontierHarness leaderboard, which analyzed 27,280 coding agent runs utilizing 83 billion tokens across various models and harnesses. This expanded version moves beyond the previous single model benchmark to include 22 harnesses tested against five models and 62 tasks, significantly increasing the scale of the evaluation. The findings revealed that 64% of inference spending was directed towards failed runs, underscoring the importance of optimizing both harness and model selections to maximize efficiency and performance.
The study indicates that different models are best paired with specific harnesses to achieve optimal cost-effectiveness, with median harness costs varying significantly across models. For example, Codex CLI emerged as the best-performing harness for GLM-5.3 but fell to 16th place when paired with Kimi-K3. The insights suggest that careful selection and pairing of harnesses with models can dramatically affect both pass rates and costs per task, driving essential conversations in the AI/ML community about resource management and system design. The evaluation also emphasizes the need for rigorous testing of trial outcomes, as most costs stem from runs that did not pass. Future steps include integrating closed models to further explore these dynamics.
Loading comments...
login to comment
loading comments...
no comments yet