LLMs Are Software Diamonds (www.evalapply.org)

The piece argues LLMs are like cut diamonds: once trained they are deterministic, high‑resolution pure functions (next‑token predictors) that refract inputs in many surprising ways, yet training itself is naturally non‑reproducible—identical data, hardware, and code will not yield identical weights across runs. Crucially, an LLM by itself is inert and stateless: it interpolates patterns from its frozen training distribution but doesn’t internally mutate or “think” in the biological/process sense. Dynamic behavior and apparent agency come from external middleware—prompts, fine‑tuning, external memories, orchestrators, API chains and human interaction—that provide state, context and composition around the model to produce emergent workflows. For practitioners this framing clarifies technical and epistemic boundaries: LLMs are powerful pattern generators that can accelerate hard problems (protein folding, proof sketching) by searching and recombining training‑derived structures, but they can’t truly extrapolate beyond their data or self‑adapt without external systems. The essay highlights implications for reproducibility, benchmarking, and system design—emphasizing development of robust orchestration, stateful augmentation, and evaluation methods—while warning against anthropomorphic overclaims about “aliveness” and urging attention to corpus biases and the human role as the critical middleware that makes LLMs useful.