LLM APIs Are a Synchronization Problem (lucumr.pocoo.org)

Working with LLMs through provider APIs exposes a deeper engineering reality: these systems behave like distributed state machines, not simple message transformers. Under the hood a model consumes token sequences, retains derived GPU state (notably the attention key/value cache), and produces next-token activations; what providers present as “messages” is only a high-level view. APIs routinely inject unseen tokens (role markers, tool defs, provider-side reasoning or search results), return opaque blobs you must echo back, and force full-history retransmission each turn. That causes both network costs (request sizes grow linearly per turn and cumulatively quadratically over a session) and compute costs (attention grows quadratically with sequence length), plus brittle server/client state splits that can diverge, get corrupted, or be unrecoverable under network partitions — as seen with server-side state features like OpenAI’s Responses API. The practical takeaway for the AI/ML community is that we need synchronization-first abstractions. Many problems map cleanly to distributed systems solutions: treat prompt history as an append-only log, KV caches as checkpointable derived state, and provider-side hidden context as replicated documents with hidden fields. Existing intermediaries and SDKs can’t fully unify incompatible hidden states across providers, so any standardization should start from model semantics (hidden state, replayability, failure modes, sync boundaries) not from current JSON message surfaces. Borrowing patterns from local-first/CRDT and checkpointing approaches would yield safer, more efficient APIs — and avoid locking in fragile abstractions that won’t scale with longer, agentic workflows.