GBench: DeepMind open-sources evaluation suite with 86 benchmark suites (github.com)

DeepMind has released gBench, an open-source benchmarking and evaluation suite designed for assessing large language models (LLMs) and vision-language models (VLMs). With 86 benchmark suites under its belt, gBench evaluates model performance across three tiers: academic evaluations, Golden Set smoke tests, and an agentic quality suite. It supports both local and remote testing via OpenAI-compatible endpoints and has been validated on platforms like vLLM and Ollama. The significance of gBench for the AI/ML community lies in its comprehensive framework that not only benchmarks model capabilities but also facilitates performance assessments tailored to specific deployment environments. Key features include rigorous academic evaluations that cover six capability pillars—such as General Knowledge, Tool Use, and Multimodal Vision—alongside deterministic Golden Set checks for model reliability. Furthermore, it automates stress testing and supports model discovery and GPU sizing, ensuring developers can efficiently optimize their AI infrastructures. This extensive suite aims to enhance the transparency and repeatability of AI evaluations, making it a vital resource for researchers and engineers working in the rapidly evolving AI landscape.