Show HN: Coder Eval – A Framework for Evals (github.com)

Coder Eval is a newly announced open-source framework designed to evaluate and benchmark AI coding agents like Claude Code, Codex, Google Antigravity, and OpenCode. The framework runs coding agents in a sandbox environment against declarative YAML tasks, providing a unique scoring mechanism that evaluates the effectiveness of coding skills and tools used by these agents, rather than simply benchmark results against fixed datasets. Key features include sandboxed execution, continuous scoring from 0.0 to 1.0, and the ability to conduct A/B testing across various models and configurations, making it easier for developers to assess the quality and reliability of coding agents in real-world scenarios. This initiative is significant for the AI/ML community as it enables developers to maintain, validate, and improve skills continuously, addressing potential "silent regressions" that can occur when models or skills change. By integrating into CI pipelines, Coder Eval allows for real-time evaluations, ensuring that coding skills remain effective as underlying AI capabilities evolve. With detailed telemetry tracking and a robust experiment layer, it empowers developers to fine-tune their models and coding agents systematically, ultimately enhancing the functionality and reliability of AI-driven coding tools.