Slow Agents (fred.glass)

They migrated from a retrieval-augmented (RAG) workflow to a fully agentic system and gained quality but saw median latency spike from 3.5s to 14s—caused by increased tool usage and a model swap to Claude Sonnet 4, which is strong at agentic tasks but has low output-token throughput. After building a high-fidelity testbed and repeating experiments to control for LLM nondeterminism and provider load, they found the core bottleneck was output tokens: reducing input tokens barely cuts latency, while shrinking output tokens produces large (~50%) latency wins. Model changes were considered (GPT-4.1 hit p50 ≈10s with minor quality drop), but switching models proved risky due to hidden “reasoning” tokens and differing agentic behavior. They evaluated three levers—faster models, concise answers, and fewer rounds—but each had tradeoffs or limited p50 impact. The fastest practical win was improving user perception: stream progress messages from tools and stream the final answer only after the model emits a wrapped <answer>...</answer> token to avoid showing intermediate “thinking.” This pragmatic fix shipped in under a week and cut median time-to-first-answer token from 14s to 8.5s, unblocking the interactive release. Longer-term plans include revisiting model swaps (GPT-5/Claude Haiku 4.5) and reducing tool-call redundancy or parallelizing/speculating tool use to further lower latency.