SimpleQA Verified: Reliable Factuality Benchmark to Measure Parametric Knowledge (arxiv.org)

A new benchmark named SimpleQA Verified has been introduced to enhance the evaluation of Large Language Models (LLMs) in terms of factual accuracy. Comprising 1,000 carefully curated prompts, this benchmark addresses several shortcomings found in OpenAI's original SimpleQA, such as noisy labeling and question redundancy. Through a rigorous filtering process that included topic balancing and source reconciliation, SimpleQA Verified offers a more robust and challenging evaluation tool. It allows researchers to better assess the factuality of LLMs while mitigating the risk of generating inaccurate responses. The significance of SimpleQA Verified lies in its capacity to track advancements in parametric models' factual accuracy, a critical aspect of ensuring reliable AI outputs. Notably, Gemini 2.5 Pro achieved a state-of-the-art F1-score of 55.6 on this benchmark, surpassing other leading models like GPT-5. This development is pivotal for the AI/ML community, providing a higher-fidelity evaluation framework that can guide future improvements in LLM reliability and reduce the prevalence of hallucinations in AI-generated content. The dataset, evaluation code, and leaderboard are readily accessible, encouraging widespread adoption and further research in this area.