Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking (arxiv.org)

A recent study titled "Do LLMs Know What to Ask and When?" investigates the ability of large language models (LLMs) to engage in multi-turn information-seeking interactions when faced with underspecified questions. The research formalizes multi-turn information-seeking as a k-underspecified constraint satisfaction problem, assessing LLMs with the MT-InfoSeek evaluation suite, which consists of 5,251 problems across various domains such as mathematics, biology, and general knowledge. The study reveals that while LLMs recognize they require additional information, they often misjudge the extent of missing details and fail to identify the minimal queries necessary for arriving at a solution. This analysis holds significant implications for the AI/ML community as it highlights the limitations of current LLM evaluations, which typically focus solely on final answer accuracy. The findings underscore the distinct skill set required for effective multi-turn information seeking, revealing that models can struggle with query ordering and still fall short even when equipped with the correct context. As researchers refine how LLMs are evaluated, this work prompts a reevaluation of their capabilities in dynamic, iterative questioning environments, which is critical for enhancing their functionality in real-world applications.