Tool Calls Are Expensive and Finite (www.reillywood.com)

Giving LLMs tool access turns them into agents with capabilities beyond text, but tool calls are far more expensive and limited than ordinary function calls. Under the hood, a tool call is just generated text: the model emits a JSON-like tool_call message, the runtime executes the corresponding function, and the result is injected back into the conversation as new tokens. That round trip consumes generation time, API tokens and — crucially — context window space. Do it once and the cost is negligible; do it hundreds of times (e.g., fetching 100 user records or adding many numbers via repeated calls) and you hit massive latency, token costs, and likely exhaust the model’s context window. This has practical implications for agent design: naive, loop-based approaches that are trivial in code become infeasible with agents. Mitigations include designing fewer, more flexible/batched tools (returning aggregated results), letting agents generate and run code where safe, or restructuring tasks to minimize round trips. Security and execution-control remain considerations if you allow code execution. In short, treat tool calls as a scarce resource — optimize for fewer, richer interactions rather than many fine-grained calls.