Intelligence per Dollar (tomtunguz.com)

Microsoft recently introduced a significant new metric, "Average Token Usage," to its AI model release card, marking a shift toward a dual benchmarking approach that evaluates both performance and cost. In a comparative analysis, Microsoft's model achieved a score of 71.6 on the SWE-Bench Verified benchmark while utilizing only a third of the tokens consumed by its competitor, Claude Haiku 4.5. This innovation answers a critical question for buyers: what is the intelligence they receive per dollar spent? As noted, even industry giants like Uber and Salesforce are facing budget constraints related to AI expenditures, with Salesforce notably spending $300 million on AI and subsequently freezing engineering hires. The introduction of this dual benchmark compels AI model developers to compete on both performance and cost-effectiveness, shifting focus from token usage to financial outcomes. As a result, companies will need to align their pricing strategies with customer perceptions of value, moving beyond traditional token-based pricing to metrics based on tangible results. This development signifies a crucial evolution in the AI/ML landscape, emphasizing the importance of efficiency in delivering high-quality intelligence while managing costs effectively.