Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing (arxiv.org)

Recent research highlights a critical issue in semantic query processing using large language models (LLMs), revealing significant run-to-run variance in query outputs. By analyzing over 1.2 million model calls across multiple benchmarks, the study found that up to 10.2% of operator outputs can change between executions, even when queries are identical and LLM plans are deterministic. This variance poses challenges for the reliability of semantic query engines, particularly as the number of rows in a dataset increases, which can exponentially amplify discrepancies in query responses. The implications for the AI/ML community are substantial, as this variation not only affects the consistency of results but also calls into question the robustness of benchmark evaluations. The findings emphasize the need for revised reporting and design rules for benchmarks and query engines, as current practices do not adequately capture the extent of variance. This research urges a reevaluation of how results are interpreted and compared within the semantic query processing landscape, impacting both academic research and practical applications in AI-driven database management.