LLMs Are Still Worst at Complex Tasks (medium.com)

Over two years of building LLM-driven agents, the author found they excel at many narrow tasks but break unpredictably as complexity rises. A concrete enterprise example: an agent was asked to produce a flat file to trigger a BizTalk map. It generated a file that looked correct—structure, fields and values all present—but running it did nothing. After multiple feedback loops the agent kept tweaking surface-level schema details and formats without resolving the failure. Tracing (via Langfuse) showed the agent had the schema and map open the whole time, yet it missed a critical nuance: the BizTalk map had elementFormDefault undefined, so every element required xmlns="" in the XML. The agent never reasoned about that implicit configuration requirement. This story highlights a core limitation of current LLM agents: strong pattern-matching and generation capability, but brittle handling of implicit constraints, cross-file dependencies and protocol semantics. For enterprise-grade automation this means failures are hard to diagnose and fix with simple prompt engineering. Technical remedies include stronger grounding (executable checks, unit tests, constraint validators), better observability/tracing, explicit symbolic reasoning or formal specs, and tighter tool orchestration so agents verify semantic invariants instead of only surface forms. Without those, scaling LLM agents into complex, safety-critical workflows will remain risky.