Beliefs that are true for regular software but false when applied to AI (boydkane.com)

Many people treat AI like traditional software — a finite codebase with bugs you can locate, patch, and prevent from recurring — but modern large language models (2025 frontier LLMs) behave very differently. Where conventional vulnerabilities usually come from mistakes in a relatively small codebase (~tens of megabytes), LLM failures mostly stem from massive, distributed training data and reward-model effects: datasets measured in thousands of gigabytes (e.g., FineWeb’s ~15 trillion tokens ≈ 11.25 trillion words) that no human can fully read or attribute. That makes root-cause debugging almost impossible, fixes non‑guaranteed (a retrain may hide a failure for tested prompts but not eliminate it), and behavior highly sensitive to tiny input changes, deployment tweaks, or previously undiscovered “emergent” capabilities discovered post‑release. For the AI/ML community this matters for safety, testing, governance, and incident response: traditional requirements/specification workflows, deterministic regression tests, and code-audit mental models don’t translate. Practical implications include prioritizing data provenance and curation, continuous red‑teaming and monitoring, robust evaluation suites (with the understanding they are incomplete), interpretability research, and public education to bridge the expert–novice gap. In short: you can’t “just patch” an LLM the way you patch software — mitigations are probabilistic and empirical, so communication, new engineering practices, and stronger assumptions about residual risk are essential.