Clever Hans Couldn't Do Arithmetic, and LLMs Don't Understand (codemanship.wordpress.com)

The article draws a clear parallel between the infamous "Clever Hans" effect and how large language models (LLMs) like GPT-4 operate, cautioning against attributing genuine understanding or intelligence to these systems. Despite their impressive ability to generate coherent and contextually relevant text, LLMs fundamentally rely on statistical pattern matching and next-token prediction rather than true comprehension or reasoning. The author recounts their own experience testing GPT-4 on a chess game, where it became evident the model neither understands the rules nor plans moves—it merely predicts likely continuations based on vast prior game data. This insight is significant for the AI/ML community because it challenges common misconceptions about LLMs’ cognitive abilities, urging users and developers to approach these tools with a critical, evidence-based mindset. Just as Clever Hans appeared to perform arithmetic by picking up on subconscious human cues, LLMs’ seeming "intelligence" is often our projection onto sophisticated but non-sentient pattern recognition. The article underscores the importance of recognizing these limitations, especially given the tendency for confirmation bias to highlight model successes while ignoring their frequent errors. The piece also introduces a practical analogy—the “brown M&Ms” test inspired by Van Halen’s concert rider—which illustrates how small, deliberate checks can reveal whether a system truly follows instructions, highlighting the need for vigilance in managing LLM deployment.