Agentic AI's OODA Loop Problem (www.schneier.com)

The piece reframes agentic AI risk through John Boyd’s OODA loop (Observe, Orient, Decide, Act), arguing that modern agents run these loops on untrusted, adversary-controlled inputs. Web-enabled LLMs, retrieval-augmented generation, tool-calling APIs and persistent state (chat history, caches) expose every stage to attacks: prompt injection, training-data poisoning, semantic backdoors, adversarial examples, and tool confusion (e.g., MCP-style interfaces). Because models mix data and control, attackers can embed malicious instructions in web content or training corpora, activate behaviors later, and propagate compromises across agents and sessions. Temporal asymmetry (poisoning during training that activates at deployment) and stateful cache poisoning make compromises long-lived and hard to audit. Technically, the problem is structural: current LLM architectures lack privilege separation or token-level authority, so there’s no reliable way to mark instructions as trusted; fine-tuning and RLHF don’t remove backdoors. The author frames a security trilemma—fast, smart, secure: pick two—warning that speed amplifies vulnerability by reducing verification time. The implication for ML practitioners is stark: “fixing hallucinations” is insufficient; we need architecture-level semantic integrity—provenance, authenticated inputs, verifiable tool semantics, audited memory—and new integrity-first designs rather than retrofitted defenses. Without rethinking boundaries and trust models, agentic AI will remain fundamentally exposed to adversaries literally inside its decision loop.