HalluHard: A Hard Multi-Turn Hallucination Benchmark (arxiv.org)

Researchers have introduced HalluHard, a challenging multi-turn hallucination benchmark designed to evaluate the groundedness of large language models (LLMs). With 950 seed questions across critical domains such as legal, medical, and coding, this benchmark aims to address the persistent issue of LLMs generating plausible but factually incorrect responses, particularly in multi-turn conversations. A novel judging pipeline allows for thorough evaluations by retrieving and parsing evidence from web sources, ensuring that claims are genuinely supported by credible citations. The significance of HalluHard lies in its ability to quantify and understand hallucination tendencies of various models, revealing that even the most advanced configurations, like Opus-4.5, still encounter approximately 30% hallucinations, despite leveraging web search for grounding. This highlights the ongoing challenges in developing models that consistently deliver factually accurate information. The findings suggest that model capacity, dialogue context, and reasoning effectiveness significantly influence hallucination behavior, paving the way for further research into reducing these errors and enhancing the reliability of AI-generated content in high-stakes applications.