ProgramBench (www.vals.ai)

ProgramBench has introduced a rigorous new benchmark that evaluates AI models' ability to reconstruct command-line programs from executable binaries and behavioral specifications. The striking metric is the "Fully Resolved" score, where models must pass all given tests to count as successful. In this initial test, top-performing models like Claude Opus 5 achieved just 6 out of 200 tasks as fully resolved, despite scoring well—82.27% on the Raw Pass Rate, which indicates their performance on hidden tests. This discrepancy highlights a crucial distinction between passing behavioral tests and fully resolving the underlying tasks, suggesting that existing models may excel at specific tasks while struggling with others due to the challenging nature of the evaluations. Significantly, the results indicate a gap in model effectiveness; while some like GPT-6 Astra boast a Raw Pass Rate of 85.4% and achieving 11 Fully Resolved tasks, others demonstrate a tendency to leverage fallback mechanisms for scoring, as seen with Claude Fable 5. This benchmark not only tests technical competencies but also drives home the point that passing hidden tests does not equate to comprehensive solution correctness. The insights provided by ProgramBench could be vital for future AI/ML research, as they encourage developers to refine their models and address these discrepancies in performance.