# One customer sends 50,000 req/min against a 100/min quota across 20 instances. How do you build the rate limiter? — retained record

> Retained reddit record from reddit:developersIndia.

Canonical: https://highsignal.app/data/records/5eb8f70010c50243

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:developersIndia)  published 2026-10-02 04:41 ingested 2026-10-03 02:37

[original source ↗](https://www.reddit.com/r/developersIndia/comments/1wvkscx/one_customer_sends_50000_reqmin_against_a_100min/)

## retained text



```
I'm a backend dev. We practice system design a lot on whiteboards, but rarely on something that looks like a real incident. Here's a production-style puzzle.

Your public API serves 50,000 paying customers, each allowed 100 requests/minute. There is currently no rate limiting at all. A bug in one customer's integration starts sending 50,000 requests/minute to `/search` (500x its quota), and latency degrades for everyone else.

**Constraints**

* 20 stateless API instances behind a round-robin load balancer, no sticky sessions
* A Redis cluster already exists (used for caching elsewhere)
* The limit must hold globally per customer. An instance-local counter lets one customer reach 100 × 20 = 2,000 req/min
* Under 5ms added to the request path at p99
* Rolling reset: no bursting 100 requests either side of a minute boundary
* Rate limiting must never be what takes the API down

**What you observe**

* The flood spreads evenly across all 20 instances, about 2,500/min each, and every instance sees a slice that looks locally plausible
* p99 latency for other customers rises in the same window as the flood
* A read-then-increment limiter passes every single-instance test, but quota is still exceeded once two instances overlap
* The limiter's Redis call is \~0.3-1ms p99 when healthy, but p99 in the seconds when Redis is degraded
* At full quota across all customers, the limiter makes \~83,000 calls/sec

**The question:** Design the limiter end to end. Where does the counter live? What makes check-and-increment indivisible? What happens when Redis is down? What changes when limits are per-endpoint?

I'd love to hear how you'd explain your design in a review, and especially what you'd do when Redis is slow or down, and why.
```



document fetched 2026-10-03 00:25 · canonical https://reddit.com/r/developersIndia/comments/1wvkscx/one_customer_sends_50000_reqmin_against_a_100min
