Our LLM cost model was double-counting batch and cache discounts (beforeyouship.dev)

A recent analysis revealed that the cost model for batch processing in large language models (LLMs) was inaccurately double-counting discounts associated with batch and cache usage. The batch API offers a straightforward 50% discount for deferred requests, but its actual savings depend on the proportion of calls that can wait. Entities were mistakenly applying both discounts additively, which could mislead businesses into anticipating greater savings than achievable. The reality is that batch pricing is multiplicative, applied only after cache savings are considered, leading to more modest discounts based on actual workloads. This clarification is crucial for the AI/ML community as it informs cost-effective strategies for utilizing LLMs. For workloads that can accept delays, such as document processing and synthetic data generation, understanding the limitations of batch discounts and optimizing caching will be key to managing expenses effectively. Future decisions regarding model tiers and the use of batch processing should prioritize task suitability and cash flow management rather than defaulting to batch-first strategies, which may impose unnecessary latency.