DatBench: Discriminative, faithful, and efficient VLM evaluations (arxiv.org)

DatBench, a new evaluation framework for vision-language models (VLMs), aims to enhance accuracy and efficiency in model assessments while significantly reducing costs associated with evaluation. The creators identified critical flaws in existing evaluation methods—such as reliance on misleading multiple-choice formats and the presence of easily solvable questions that do not benefit downstream applications. By transforming these elements into generative tasks and filtering out low-quality samples, DatBench demonstrates a marked improvement in discriminability and faithfulness, achieving a dramatic average speedup of 13 times (and up to 50 times) compared to traditional evaluation datasets. This development is significant for the AI/ML community as it highlights the urgent need for better evaluation practices that keep pace with the rapid advancements in VLM technology. By addressing key issues like evaluation fidelity and computational cost—where up to 20% of development compute is historically dedicated to evaluations—DatBench paves the way for more rigorous and sustainable evaluation methods. Released alongside DatBench-Full, which offers a comprehensive cleaning of 33 datasets across nine VLM capabilities, this initiative is poised to guide researchers toward more effective assessment strategies in an evolving landscape of machine learning models.