Do LLMs Reason, or Do They Just Predict Math Text? (daridor.blog)

A recent analysis of the KisMATH dataset, accepted for publication in TACL, challenges the claim that large language models (LLMs) such as those developed by OpenAI can "internally realize" mathematical reasoning structures. Although the dataset itself is significant — comprising 1,671 problems from established benchmarks like GSM8K and MATH500 — the findings that certain math tokens receive higher next-token probabilities than random tokens are framed as evidence of reasoning, leading to inflated interpretative claims. The actual experiments suggest that LLMs are simply good at predicting mathematical continuations in context, rather than demonstrating genuine reasoning capabilities. The study reveals methodological shortcomings, particularly with its focus on how LLMs perform across varying complexities of math problems. For simpler tasks, the evidence supports the hypothesis, but for more challenging olympiad-level problems, the data shows that linguistic elements — not just mathematical structures — also contribute significantly to model performance. Critics argue that the framing of the results conflates observations with stronger interpretations of internal cognitive mechanisms, suggesting that further empirical work is needed to delineate the actual capabilities of LLMs in reasoning tasks. This calls into question not just the current findings, but also future applications of LLMs in domains heavily reliant on reasoning.