AI interpretability has the same problems as philosophy of mind (www.seangoedecke.com)

Researchers use “probes” — classifiers or learned sparse encoders applied to a model’s layer activations — to surface neuron-level “concepts” as part of mechanistic interpretability. The piece argues that probes have an intrinsic epistemic problem: very powerful probes can read complex patterns out of a model’s activations even when the model itself doesn’t use those concepts (e.g., a GPT‑4 probe finding “second‑order belief” patterns in a tiny model that can’t reason about beliefs). That mismatch is like a linguist detecting German‑verb patterns in a non‑German speaker’s brain: the information is present in the encoding, but not necessarily in the system doing the processing. Practically, this means naive probe results can be spurious unless backed by causal tests — boosting or ablating the pattern and observing behavior (Anthropic’s Golden Gate Claude is cited as a gold‑standard demo). The author ties this to philosophy of mind (Dennett’s “Real Patterns”) and outlines rival stances — from literal realism to eliminative materialism — to show the debate isn’t new: are concepts “in” a brain or just useful descriptions? The takeaway for AI/ML: mechanistic interpretability is valuable and complements “folk” AI psychology (talking about beliefs, personalities, concepts), but researchers should be skeptical, restrict probe power, and insist on intervention-based validation before claiming a discovered feature is genuinely used by the model.