Tuning the Harness, Not the Model (www.langchain.com)

In a recent advancement, the Nemotron 3 Ultra model achieved near-frontier agent performance, with a score of 0.86 on the Deep Agents benchmarking suite, closely trailing the leading Opus 4.8 at 0.87. This was accomplished by focusing solely on tuning the agent's harness—the system prompt, tool descriptions, and middleware—rather than modifying the underlying model itself. This approach resulted in a dramatic cost reduction, with the evaluation cost per run reduced to approximately $4.48 compared to Opus 4.8's $43.48, while maintaining low latency. This indicates that significant performance gains can be achieved without the need for costly and resource-intensive model training. The significance of this development lies in its potential to democratize access to advanced AI capabilities. By utilizing open models, the AI community can achieve results comparable to top-tier models at a fraction of the cost. The findings highlight the importance of harness engineering in maximizing model performance—showing that when a harness is well-tuned to an agent, it allows the model to effectively leverage its capabilities. The study utilized a methodical, data-driven approach to refine the harness iteratively, emphasizing that improvements were secured through empirical testing rather than speculative changes. This paves the way for more optimized and cost-effective deployments of AI agents across various applications.