Throughput vs. Goodput: The Performance Metricin LLM Testing (qainsights.com)

A recent blog post highlights the crucial distinction between throughput and goodput in the context of testing large language model (LLM) deployments. Throughput, a long-standing metric, measures how much work a system can process within a time frame without considering quality or latency, potentially leading to a false sense of confidence. In contrast, goodput, as defined by NVIDIA’s AIPerf tool, reflects the number of completed requests that meet specific service level objectives (SLOs), specifically focusing on latency metrics such as Time to First Token (TTFT) and Inter-Token Latency (ITL). This distinction is vital because it directly impacts user experience, where high throughput can mask significant performance degradation, leaving users frustrated with slow response times. The blog emphasizes that goodput provides a more accurate measure of whether LLM systems are effectively serving users, particularly under load. It explains how throughput can appear stable while goodput declines, indicating that many requests fail to meet qualitative standards. By implementing goodput checks alongside traditional throughput metrics, developers can ensure that their LLM applications are not only operational but also performant and user-friendly. This approach encourages a more thorough evaluation of system readiness for production, particularly as AI/ML applications scale to meet increasing user demands.