OpenBench: Provider-agnostic, open-source evaluation infrastructure for LLMs (github.com)

OpenBench is an open-source, provider-agnostic benchmarking infrastructure for large language models released as an alpha build and being developed in public. It provides a standardized, reproducible CLI-driven workflow (bench eval/list/describe/view) for running 35+ benchmark suites — including MMLU, GPQA, HumanEval, competition math (AIME/HMMT), SciCode, GraphWalks, long-context retrieval, and HealthBench — against models from 30+ providers (Groq, OpenAI, Anthropic, Google, Cohere, Hugging Face, AWS/Azure, Ollama/vLLM local, etc.). Significance: by unifying diverse benchmarks and providers under one extensible framework, OpenBench makes cross-provider comparisons, private/local evaluations, and community-driven benchmark development far easier and more reproducible for researchers, evaluators, and product teams. Technically, OpenBench builds on Inspect AI, exposes model/task-specific flags (-M, -T) and standard options (--limit, --temperature, --max-connections, --epochs, --timeout), and supports plugin benchmarks via Python entry points so suites can iterate independently. It supports grader models for certain benchmarks (requires an extra API key), Hugging Face Hub export of results, and explicit sandboxing/security guidance for risky plugins (e.g., openbench-cyber). Local/private evals are first-class (bench eval <path>), cache and logs live under ~/.openbench, and providers are wired via simple environment variables—making it straightforward to run reproducible, multi-provider evaluations or drop-in custom benchmarks into CI and research pipelines.