Without benchmarking LLMs, you're likely overpaying 5-10x (karllorey.com)

A recent analysis revealed that many businesses utilizing large language models (LLMs) can save substantially—up to 80%—by properly benchmarking models for their specific use cases. Initially, a non-technical founder faced escalating costs while using GPT-5, a common choice due to its familiarity and established benchmarks. However, a detailed benchmarking process showed that GPT-5 was often not the most cost-effective option, despite its strong performance metrics. By assessing 100+ alternative models against tailored tasks—like customer support queries—the founder was able to identify cheaper models that provided comparable, and sometimes superior, output, ultimately reducing expenses by over $1,000 monthly. The study highlights the critical importance of tailored evaluations when selecting LLMs, as generic benchmarks may not accurately predict performance for specific applications. A structured approach was employed, building custom benchmarks based on real examples, specifying expected outputs, and applying scoring methodologies using an LLM as a judge. This exercise revealed that even models perceived as best-in-class could be outperformed in terms of cost and quality by less common alternatives. To facilitate ongoing optimization, a new tool called Evalry was developed, allowing users to benchmark their unique prompts across over 300 LLMs, streamlining the process of identifying the best models for both quality and cost-effectiveness.