Making agents fast – what I've learnt over the last 12 months (martinalderson.com)

In a recent reflection on optimizing AI agents, a developer shared insights gained over the past year, focusing primarily on improving inference speed. Key metrics for this are "time to first token" (TTFT) and "tokens per second" (tok/s), which affect how quickly responses are generated from language models. Understanding the nuances of these metrics is crucial; for example, in workflows requiring frequent short responses, TTFT becomes more critical than tok/s, while the opposite is true for longer outputs. The developer also emphasized the importance of prompt caching and the variability in speed across different inference providers, highlighting the need to compare models under similar conditions. Beyond inference speed, the developer underscored the role of classic performance optimizations, such as analyzing tool call traces and refining the underlying hardware. Identifying slow command executions and optimizing resource usage—like managing RAM during suspend-resume cycles—can lead to significant improvements. The insights underline that while faster inference is beneficial, comprehensive system optimization, including providing agents with the right tools and ensuring robust hardware capabilities, remains essential for enhancing overall efficiency in AI workflows.