# Our p99 latency dropped about 40% after we stopped setting CPU limits on most workloads. Curious what others have seen — retained record

> Retained reddit record from reddit:kubernetes.

Canonical: https://highsignal.app/data/records/db80d9dd32c4329f

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:kubernetes)  published 2026-10-14 02:36 ingested 2026-10-15 02:41

[original source ↗](https://www.reddit.com/r/kubernetes/comments/1x5fwo2/our_p99_latency_dropped_about_40_after_we_stopped/)

## retained text



```
We spent a few weeks chasing intermittent latency spikes on a Node.js service running on EKS. CPU usage looked fine in Grafana, averaging around 35% of the limit, but p99 kept spiking under bursty load.

The culprit turned out to be CFS throttling. With a 500m limit, the container gets 50ms of CPU time per 100ms period. A burst of work in the first 20ms can burn through that quota, and the process then sits frozen for the rest of the period, even while the node has idle cores. Averaged metrics hide this completely. The metric that actually showed it was `container_cpu_cfs_throttled_periods_total` divided by `container_cpu_cfs_periods_total`. Ours was throttling in over 30% of periods while "average CPU" looked healthy.

What we changed:

* Kept CPU requests accurate, since they drive scheduling and cgroup shares
* Removed CPU limits on latency-sensitive services, but kept memory limits
* Added an alert on throttle ratio instead of just raw usage

Tradeoffs we're still working through: noisy neighbors on shared nodes, how this interacts with the static CPU manager policy for guaranteed pods, and whether it's safe in multi-tenant clusters where you can't trust other teams to set honest requests.

Full breakdown and the PromQL queries here: [https://docs.google.com/document/d/1RXUNIQ5\_\_E\_H5YvYKAhJtEsPsrNLjU1VRBMEDbZvku0](https://docs.google.com/document/d/1RXUNIQ5__E_H5YvYKAhJtEsPsrNLjU1VRBMEDbZvku0) (disclosure: this is my own writeup, no product or paid service involved)

It's most relevant if you run latency-sensitive services and use CPU utilization dashboards to judge headroom.

Has anyone gone the other way and found limits worth keeping? I'm especially interested in how people handle this in multi-tenant setups.
```



document fetched 2026-10-15 00:40 · canonical https://reddit.com/r/kubernetes/comments/1x5fwo2/our_p99_latency_dropped_about_40_after_we_stopped
