Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structures (arxiv.org)

Researchers have introduced a novel benchmark called DSR-Bench (Data Structure Reasoning Benchmark) aimed at evaluating the algorithmic reasoning capabilities of large language models (LLMs). This benchmark focuses on the manipulation and understanding of data structures, which are essential for performing complex, multi-step decision-making tasks in computational contexts. With 20 data structures, 35 operations, and over 4,140 problem instances, DSR-Bench provides a structured framework for assessing LLMs' abilities to reason about order, hierarchy, and connectivity—key elements of effective algorithmic reasoning. The significance of this development lies in its capacity to uncover critical limitations within leading LLMs. In evaluations, even the best-performing model scored only 0.46 out of 1 on challenging tasks, highlighting substantial weaknesses in reasoning with spatial data and context-rich scenarios, as well as difficulties in reasoning about their own code. As LLMs are increasingly relied upon for complex applications, the insights from DSR-Bench could guide future improvements and research in the AI/ML community, emphasizing the need for models that can competently tackle structural reasoning tasks.