Cutting our agent's harness cost 10x without breaking the product (www.usenotch.ai)

Notch, an AI startup specializing in video ad generation, achieved a staggering 90% reduction in its agent's harness costs, dropping from $4.44 to $0.50 per session, by switching from the Sonnet-powered harness to GPT-5.6 Luna. The decision to transition was driven by the realization that a significant portion of their AI expenditure was tied to harness coordination. Notch discovered that their existing SDK, the Claude Agent SDK, allowed for seamless integration with different models through the Anthropic Messages API, enabling them to leverage cheaper alternatives without overhauling their entire system. This shift is significant for the AI/ML community as it demonstrates the potential for cost savings while maintaining functional output quality. The team highlighted the importance of bench-testing models based on real-world workloads rather than solely relying on popular benchmarks, discovering that models can exhibit vastly different performance in practical scenarios. Key lessons included the critical need to evaluate the full cost of deploying a model, considering factors such as retries and regeneration, to truly assess economic benefits. This case illustrates that while the initial token price may be appealing, the overall economic viability relies heavily on the model's operational synergy with existing tools and its ability to produce effective outcomes consistently.