Psycholinguistics and Transformer Circuits (nathanzhao.cc)

Nathan’s Notes maps classic psycholinguistic findings onto what we can now see inside transformer “circuits,” showing striking parallels and crucial differences. Using the lexical decision task as a touchstone, the piece reviews facilitation (semantic priming) and inhibition (inhibitory priming, neighborhood-density effects), the Cohort Model’s parallel activation of candidates, and electrophysiological markers (N400/P300) that index semantic surprise and decision uncertainty—analogous to high-entropy token distributions in LLMs. Eye-tracking and TRACE-model results (e.g., rhyme competitors attracting fixation) show phonological activation is emergent and time-dependent, not a simple left-to-right lookup—an emergent property transformers can mimic in part despite only seeing text. Technically, the essay contrasts biological spreading activation—time delays, decay, fixed-capacity state—with transformers’ growing context windows and non-decaying access mediated by attention weights. It notes transformer circuits can learn phonetic-like patterns from orthography and structured data (poems, rhyme schemes), but lack intrinsic phonetic representations and temporal dynamics that underpin human inhibitory priming and recency effects. For AI/ML this suggests concrete research directions: integrate multimodal/phonetic inputs, build self-contained memory with decay or compressive updates, and model realistic temporal propagation to better capture human-like competition and recency—potentially improving multilingual, phonological, and uncertainty-sensitive language behaviors.