Strong Model First or Weak Model First? A Cost Study for Multi-Step LLM Agents (llm-spec.pages.dev)

A new study examines two strategies for cost-effective multi-step workflows in large language model (LLM) agents that generate code: the "strong model first" approach, which uses high-quality generation upfront to minimize subsequent bug fixes, and the "weak model first" strategy, which opts for cheaper generation followed by a robust model to address errors. The research analyzes the cost implications of each strategy, taking into account factors such as bug fix probabilities, the accumulation of context during interactions, and the effective retry rates per bug. It reveals that using a fresh context per bug significantly reduces costs compared to a shared context model, which incurs higher penalties due to accumulated complexity. This study is significant for the AI/ML community as it adds a nuanced understanding of model routing and bug fixing in LLMs. By incorporating the context growth penalty—often overlooked in previous work—the authors highlight the intricacies of cost management associated with iterative corrections. The findings suggest that well-designed agents that reset context between bug fixes can greatly enhance efficiency, thereby influencing future designs and implementations of AI coding assistants. This could pave the way for new insights into AI system architectures that balance performance with budgetary constraints, optimizing both costs and outcomes in AI applications.