CNCF: Goodput Metric Reveals Nearly 10× Worse p95 Latency in Throughput-Optimized LLM Serving

What is goodput and why does it differ from throughput?

Goodput measures the number of completed requests per second that meet a predefined target latency, while throughput counts all processed requests regardless of whether they are fast enough. Akamas engineer Graziano Casto shows on the CNCF blog that higher throughput does not necessarily mean better service — a configuration may process more requests overall, but some of them with unacceptably long latency for the end user.

Benchmark: 50% higher throughput with nearly 10× worse latency

A test on the Qwen2.5-7B model, on a single NVIDIA A10G GPU within an EKS cluster (Amazon Elastic Kubernetes Service), using the vLLM serving engine and the GuideLLM load-simulation tool, showed that a configuration with 50% higher combined throughput can have a p95 time-per-output-token — the latency below which 95% of requests fall — nearly 10× higher, from about 50 milliseconds to about 494 milliseconds. Three workload types were tested: chatbot with a strict time-to-first-token (TTFT) requirement, reasoning with an output of about 4,000 tokens, and agentic with short chained calls.

Recommendation: automated parameter search

Casto recommends automated search over parameters such as gpu_memory_utilization, max_num_batched_tokens, and max_num_seqs to optimize serving configuration directly for goodput rather than raw throughput. The approach requires testing separately per workload type, since chatbot, reasoning, and agentic scenarios have different target latencies and tolerances, so no single universal configuration covers all cases equally well.

Manually searching parameter combinations for every model-and-GPU pairing is impractical given the large number of possible settings, so Casto advocates an automated process that systematically tests configurations and selects the one with the best goodput for the target workload type. Teams serving LLMs in production thus get a concrete method for avoiding the trap of optimizing solely for throughput, which in the benchmark leads to nearly tenfold worse p95 latency for the end user.