Inference cost at scale with napkin math (injuly.in)

A recent analysis has provided valuable insights into the cost of running AI models at scale, particularly emphasizing the mathematical breakdown of inference costs associated with GPU usage. By assuming a 32-billion-parameter model and leveraging essential hardware specifications like memory bandwidth and peak throughput, the study highlights that each token generated can require trillions of floating-point operations and substantial memory accesses. The optimization techniques, notably the use of KV caching, significantly reduce the computational cost by enabling models to only process the most recently generated output, thus making the setup more efficient for servicing multiple users simultaneously. For the AI/ML community, this information is crucial as it outlines practical strategies for optimizing model performance and resource allocation. The analysis suggests that while peak theoretical capacities could allow for about 300-800 users on a single GPU, actual performance will vary based on user engagement patterns and context length, with potential configurations leading to effective support for numerous concurrent chat users. Overall, this study not only demystifies the complexities of model inference but also serves as a guide for developers looking to maximize efficiency and scalability in AI applications.