Benchmarking Claude Code, Codex and Pi on SWE-Bench Pro: Same Accuracy, 2x Cost (aistack.imec-int.com)

Recent benchmarking of AI agent harnesses—specifically Claude Code, Codex, and Pi—on coding tasks revealed that while overall accuracy remains consistent across these models, the choice of harness can significantly affect operational costs. Testing on the SWE-Bench Pro dataset showed resolve rates hover between 44% to 53%, indicating that the harness impacts task resolution less than expected. However, discrepancies emerged in token efficiency costs: switching from Codex to Claude Code doubled token expenses from $0.35 to $0.70 per resolved task, potentially inflating budgets for large-scale deployments. The research highlighted technical aspects such as input and output token volume, revealing that higher context inputs did not equate to better task resolution but increased computational costs and GPU memory pressure. Notably, anomalies like ‘doom looping’ and prefix caching errors were identified as significant factors that can further drive up costs. Overall, the study emphasizes the importance of selecting the appropriate harness and model configuration to optimize both performance and budget in AI development environments.