XYBench: Helping with the Problem Behind the Question (huggingface.co)

Researchers have introduced XYBench, a novel benchmark designed to evaluate AI assistants on their ability to address misconceptions in user queries. This resource, recognized with the Best Resource Paper Award at EMNLP 2026, comprises over 8,000 questions that highlight the XY problem—where users ask for help based on incorrect assumptions. For instance, a user may inquire about thickening frosting while overlooking that the real issue lies in the temperature of the cupcakes. The benchmark emphasizes not just providing answers to users' stated requests but also recognizing and steering them toward their actual goals, which is critical for enhancing AI’s troubleshooting capabilities. Developed collaboratively by researchers from Carnegie Mellon, NVIDIA, and Johns Hopkins, XYBench exposes a significant shortfall in large language models (LLMs) when addressing user misconceptions. Evaluations of top models—such as Claude Sonnet 4.6, GPT-5, and Gemini 3.1 Pro—revealed that while they often suggested solutions, they predominantly focused on the original, flawed approach. The benchmark measures three key aspects: whether responses contain pragmatic solutions, their emphasis on addressing user misconceptions, and their ability to redirect effectively toward the true goal. By enabling researchers to pinpoint these gaps, XYBench aims to guide future AI enhancements, making assistants more adept at delivering practical advice, especially when users are unaware of their misconceptions.