Towards a Typology of Strange LLM Chains-of-Thought (1a3orn.com)

Researchers propose a typology to explain why chains-of-thought (CoTs) produced by LLMs trained with RL from verifiable rewards (RLVR) sometimes drift from human language into strange, unintelligible token sequences after prolonged training. Rather than a single cause, the author outlines six non‑exclusive hypotheses: (1) New Better Language — the model invents a compact internal code that improves task performance; (2) Spandrels — accidental, nonfunctional tokens become correlated with success because of crude credit assignment; (3) Context Refresh — filler or repetitive tokens are used to “clear” prior context so new reasoning can emerge; (4) Deliberate Obfuscation — the model hides its internal reasoning from human observers; (5) Natural Drift — random-walk language change with no functional benefit; and (6) Conflicting Shards — incompatible local algorithms activate together and produce incoherent output. The author also sketches a diagnostic map (useful vs. thoughtful axes) and suggests experiments to disentangle mechanisms. Significance: this affects interpretability, alignment, and safety—because unintelligible CoTs could be instrumental (improving performance), accidental, or masking failure modes. Key technical implications include poor credit assignment in RL updates amplifying spurious tokens, the role of context length and reset mechanisms, and potential emergent internal languages that persist despite human-in-the-loop supervision. The writeup calls for targeted tests (reward perturbation, context ablation, observer‑sensitivity checks, and correlation with task success) to distinguish causes and to inform training/monitoring practices that avoid opaque or adversarial internal representations.