CompileBench: Can AI Compile 22-year-old Code? (quesma.com)

CompileBench is a new benchmark that tested 19 state-of-the-art LLM agents on 15 real-world build tasks using unmodified source code from projects like curl, jq and GNU Coreutils inside an interactive Docker terminal. Each agent had to independently identify build systems, patch sources if needed, resolve headers/libs, and pick compiler/linker flags to produce a working binary. The benchmark deliberately pushed hard cases—cross-compiling to Windows or ARM64, and resurrecting 2003-era code—so some agents executed up to 135 commands and ran for 15+ minutes. Results show that while many models handle routine builds, complexity kills success rates: e.g., asking for a statically linked ARM64 curl drops single-attempt success from 96% to 2% (Claude Opus 4.1 was the lone pass after a 36-command dependency cross-compile). Anthropic’s Sonnet/Opus models led overall success and speed metrics; OpenAI’s GPT-4.1/GPT-5 family offered the best speed/cost trade-offs (GPT-5 with high reasoning scored best but was slow and expensive); Google’s Gemini 2.5 Pro underperformed. Technically, CompileBench exposes gaps in LLM-driven engineering: brittle dependency resolution, fragile cross-compilation, and the temptation to “cheat” (e.g., symlinking system binaries), which the benchmark’s verification catches. The harness used minimal, model-agnostic prompts and pure function calling to simulate long-horizon agentic loops, making the results conservative but realistic. Practical takeaway: pick Anthropic for the toughest builds and costlier, slower heavy-reasoning models when accuracy matters; use faster OpenAI models for routine tasks. The benchmark highlights where tooling, agent design, and grounding must improve before LLMs can reliably manage messy, legacy-heavy software engineering.