# Why GPU Utilization Is the Wrong North Star for Production AI Inference — retained record

> Retained news record from news:hpcwire.

Canonical: https://highsignal.app/data/records/02807aecbc4d1ec5

[← sources](https://highsignal.app/data)


source: [news](https://highsignal.app/data/news) (news:hpcwire)  published 2026-07-21 16:39 ingested 2026-07-22 08:32

[original source ↗](https://www.hpcwire.com/2026/07/21/why-gpu-utilization-is-the-wrong-north-star-for-production-ai-inference/)

## retained text



```
New Open Weight AI Models from China Renew Calls for Regulation
We’ve seen the release of two highly capable AI models from China over the past...
It’s 2 AM and the on-call engineer pulls up the dashboard. GPU utilization sits at 87%. No alerts have fired. The cluster looks busy, which usually means the cluster looks healthy. She closes the laptop. What she didn’t see: p99 latency on the chat completion endpoint had been climbing for four hours. At 9 PM it was 1.2 seconds. By midnight it was 6.4. Users in time zones where it was still business hours had been waiting seconds for first tokens to appear. Clients had started timing out requests. The system was failing people quietly and continuously, and the primary inference dashboard showed green.
The metric said 87% occupied. It said nothing about what the silicon was doing, how long requests were waiting, or whether any of them were being served within the latency envelope users actually experience. GPU utilization is a hardware metric. Nobody designed it to answer reliability questions. Treating it as your primary inference service level objective (SLO) is like checking engine RPM to decide whether passengers arrived on time.
The reason high GPU utilization can indicate distress rather than health comes down to what the metric actually counts: active compute cycles on the silicon, regardless of whether that work is serving requests efficiently.
In transformer inference, KV cache pressure is one of the most common culprits. As context lengths grow and concurrent requests multiply, the KV cache fills. The serving runtime responds by increasing batch sizes to amortize memory overhead, which drives up compute occupancy. GPU utilization climbs toward 95%. Meanwhile, each individual request is waiting longer in the queue to be included in a batch, and once in the batch, it’s competing for cache slots. You can hit 95% GPU utilization while median TTFT has tripled, because the hardware is genuinely busy doing work that isn’t moving requests forward quickly.
Request batching pressure compounds this. Continuous batching strategies maximize throughput at the cost of queuing latency when the system is saturated. A request arriving at peak load may sit in the inference gateway queue for 300-800ms before a single token generates. That wait is invisible to a GPU utilization monitor. The GPU wasn’t idle during that window. It was running other requests.
Utilization measures silicon occupancy. It does not measure request health, queue depth, or whether your p95 TTFT is inside any number a product manager would consider acceptable.
Three metrics replace GPU% as the operational signal for inference systems.
Time-to-first-token (TTFT) is the primary user-facing SLO for interactive workloads. It captures exactly what users feel: how long until something starts happening. TTFT degrades before throughput collapses, which makes it an early warning signal rather than a post-mortem one. Set hard p95 and p99 thresholds. Alert on them. For many interactive endpoints, a p99 TTFT above a few seconds is already a user-facing reliability problem, whether GPU utilization is 40% or 95%.
Token throughput per SLO tier replaces aggregate tokens-per-second as a throughput signal. Aggregate TPS tells you the system is producing tokens. Tiered throughput tells you whether the right classes of requests are meeting their latency envelopes. A batch summarization job consuming 60% of throughput while interactive requests queue behind it looks fine on aggregate TPS and catastrophic on tiered throughput. The distinction only appears when you track them separately.
Request queue depth at the inference gateway is the leading indicator that separates the other two. Queue depth rising before TTFT degrades is your earliest signal that saturation is approaching. If queue depth is growing and TTFT is still within bounds, you have a short window to act. If you only discover the problem when TTFT spikes, you are already serving degraded traffic.
Multi-step agentic workflows multiply every failure mode described above. A single user request that invokes a tool-calling chain or an MCP orchestration layer doesn’t generate one inference call. It generates four, or seven, or twelve, depending on how many steps the agent takes before returning a result.
GPU utilization aggregates across all of those sub-calls. It tells you the cluster is busy. It doesn’t tell you that step three in the agent chain has a TTFT of 4 seconds because it generates a long context that’s saturating the KV cache, while steps one, two, and four are completing in 300ms. The aggregate utilization number hides the bottleneck entirely.
In an agentic system, latency compounds. A five-step chain where one step degrades from 400ms to 3 seconds adds 2.6 seconds to every user request. Users don’t see individual step latencies. They see the wall clock time from request to final response, and that number reflects the worst-performing link in the chain. GPU utilization will never show you which link that is.
The fix is not more GPU dashboards. It is different instrumentation at the gateway layer. Three changes make inference systems easier to operate and harder to fail silently.
Circuit breakers at the inference gateway should key on TTFT percentiles, not GPU%. When p95 TTFT on an endpoint crosses a threshold, stop routing new requests there and either fail fast or route to a fallback tier. A GPU-keyed circuit breaker would never trip during KV cache saturation. A TTFT-keyed one trips exactly when users are already experiencing degradation.
Retry budgets should scope per agent step rather than globally. A global retry budget in an agentic system allows a single degraded step to consume all retries, leaving no budget for transient failures elsewhere in the chain. Per-step budgets contain the blast radius. Step three gets two retries. If it fails both, the chain fails fast rather than exhausting the system’s retry capacity on one bad step.
SLO tiers by inference class matter because interactive requests, batch workloads, and background agents have genuinely different latency requirements. Some interactive chat experiences may need sub-second TTFT before users start noticing friction. A background document summarization agent can tolerate 5 seconds. Treating them identically means either over-provisioning for batch or under-serving interactive users. Define the tiers explicitly, enforce them at the gateway, and alert on each independently.
GPU utilization belongs in your capacity planning workflow and your procurement conversations. It will tell you when to order more hardware. It will not tell you whether the hardware you have is serving users well. Those are different questions, and confusing them will get you paged at 2 AM over a dashboard that says everything is fine.
About the Author: Vasuki Uday Kiran Vudathala is a Staff Performance Engineer at ServiceNow with over 15 years of experience in performance engineering,
platform engineering, and observability for cloud-native and Generative AI platforms. He writes about AI infrastructure, platform reliability, and production system behavior, with technical articles published in InfoQ, SD Times, Stickyminds, HackerNoon and Medium.
```



document fetched 2026-07-22 08:32 · canonical https://hpcwire.com/2026/07/21/why-gpu-utilization-is-the-wrong-north-star-for-production-ai-inference