LLMs Are Weird, Man (surfingcomplexity.blog)

The author argues that large language models (LLMs) feel like “magic” because, unlike most engineered systems, their high‑level cognitive behaviors aren’t understood even by their creators. We can explain implementation details—words as vectors, training via token‑prediction on massive datasets, and the importance of scale and architecture—but we don’t know how concepts (e.g., “the number two”) or reasoning emerge from those mechanisms. Anthropic’s recent work (“Tracing the thoughts of a large language model”) exemplifies this gap: engineers are effectively doing “AI biology,” using interpretable “replacement models” and other indirect techniques to reverse‑engineer Claude’s internal mechanisms because direct explanations are lacking. That epistemic gap matters for the AI/ML community because it changes how we study, deploy, and govern these systems. Technical implications include a growing need for mechanistic interpretability methods, caution around system prompts (natural‑language configuration that surprisingly shapes behavior), and uncertainty about whether future architecture changes will yield incremental gains or paradigm shifts. Economically and socially, hype and backlash may obscure real risks and capabilities; practical experience already shows LLMs can beat search in some tasks but remain inconsistent. The takeaway: LLMs are a new kind of artifact—powerful and poorly understood—and the community should expect surprises while investing in deeper interpretability and robust evaluation.