Oil and Water: LLMs in Low‑Tolerance Workflows (www.gojiberries.io)

Adding an LLM to a deterministic workflow shifts both upside and downside: ΔValue = ΔBenefit − N Σ f_i · Δp_i · L_i − ΔCostOfControls. The note models workflows as Mealy machines (states S, inputs Σ, outputs Λ, transition δ, output λ) and shows that inserting a stochastic LLM turns formerly verifiable transitions into probabilistic ones. That creates three classes of failures—input (ambiguous NL → wrong symbol), control‑flow (non‑deterministic routing, unexpected retries or parallelizations), and output (verbalized promises or unauthorized disclosures)—which can dominate value loss because increased failure probabilities Δp_i multiply costly real‑world effects L_i. Practical mitigations are operational and architectural: keep stochastic components off commit edges (two‑phase commit + HITL for commits), render explanations from deterministic decision artifacts, enforce blast‑radius constraints (runtime invariants like “total refunded ≤ total paid”), staged rollouts, monitoring for tail risk (Σ p_i L_i, CVaR), kill‑switches, and budget explicitly for monitoring/fallbacks. Task granularity follows a “Goldilocks” tradeoff: per‑step error e(s)=c s^α + ε0 implies an analytic optimum s* ∝ (ε0/(c(α−1)))^(1/α), but adding context overhead e(s)=c s^α + ε_fixed + ε_context s^β (0<β<1) shifts s* larger and requires numerical calibration. Bottom line: LLMs add value but demand risk budgeting, continuous red‑teaming, versioned prompts, and business‑level tail metrics rather than traditional unit tests.