Cache hit rates of Inference are more meaningful than the headline costs (dirac.run)

Recent analysis highlights the critical role of cache hit rates in optimizing AI model inference costs, revealing that these metrics are often overlooked in favor of headline pricing. The study of over 60 AI providers, utilizing 398 data points from OpenRouter.ai, shows that while the price per million input tokens is important, the actual expenses incurred in multi-turn conversations can escalate significantly if cache hit rates are low. As conversation context grows quadratically with each turn, inefficient caching can result in processing costs up to ten times higher for new tokens. Among the findings, DeepSeek emerged as the leader in caching performance, boasting an impressive cache hit rate of 87%, while several US providers, including Google, fell behind in this metric. The implications are profound for developers and organizations relying on AI chatbots or agents, as subpar caching can inflate operational costs dramatically. This insight encourages the community to prioritize cache hit rates as a key performance indicator, as selecting models with high caching capabilities could lead to substantial long-term savings and better operational efficiency in AI applications.