Are published ANN-Benchmarks DBMS results trustworthy? (blog.ydb.tech)

Recent scrutiny of ANN-benchmarks, a widely used tool for evaluating Approximate Nearest Neighbor (ANN) search algorithms across various database management systems (DBMS), has raised significant questions about the reliability of its published results. As ANN search plays a vital role in enhancing AI capabilities, especially in applications like Retrieval-Augmented Generation (RAG) for large language models (LLMs), accurate performance comparisons are critical. The analysis reveals that flaws in benchmarking methodology—specifically regarding how Queries Per Second (QPS) are calculated—can lead to misleadingly low performance assessments. By addressing these methodological issues, a new fork of the benchmark showed nearly a 20-fold increase in QPS for the pgvector implementation, indicating that the discrepancies were largely due to client-side measurement errors rather than actual database performance. This finding is significant for the AI/ML community as it underscores the need for rigorous testing protocols and accurate benchmarking tools. With many DBMS now providing ANN capabilities, unreliable benchmarks could lead researchers and practitioners to make poor decisions when selecting databases for AI workloads. Improved methodologies that correctly account for concurrency and execution overheads are essential to ensure fair comparisons. As the community awaits further advancements in benchmarking techniques, this critical examination encourages developers to refine tools like ANN-benchmarks and to push for better data generation practices to enhance performance evaluations across different ANN algorithms.