Do AI Reasoning Models Abstract and Reason Like Humans? (aiguide.substack.com)

Researchers analyzed whether state-of-the-art reasoning models truly form humanlike abstractions or just learn dataset shortcuts by evaluating OpenAI’s o3, Anthropic’s Claude Sonnet 4, and Google’s Gemini 2.5 Pro on 480 ConceptARC tasks (derived from ARC/ARC-AGI-1). Models received either textual inputs (grids as integer matrices with 0–9 color codes) or visual inputs and were prompted to produce both the output grid and a natural-language transformation rule. Every generated rule was manually labeled as “correct as intended,” “correct but unintended,” or “incorrect,” and compared to human rules on concept-isolating tasks (e.g., “horizontal vs. vertical,” “complete the shape,” “largest hollow rectangle”). The core finding: with textual inputs the models often match or exceed human accuracy but frequently succeed for the wrong reasons—pixel- and feature-based heuristics rather than object-based abstractions—whereas with visual inputs they produce fewer accurate grids but sometimes generate more genuinely intended rules. Models tended to describe grids in terms of rows/columns/pixels rather than objects, revealing weak “objectness” priors and a propensity for brittle shortcuts that undermine generalization and explainability. The paper argues that accuracy alone misleads evaluations (overestimating abstraction on textual tasks, underestimating it on visual ones) and calls for rule-level and explanation-focused benchmarks to assess trustworthy, human-aligned reasoning.