One customer sends 50,000 req/min against a 100/min quota across 20 instances. How do you build the rate limiter?
retained text
I'm a backend dev. We practice system design a lot on whiteboards, but rarely on something that looks like a real incident. Here's a production-style puzzle. Your public API serves 50,000 paying customers, each allowed 100 requests/minute. There is currently no rate limiting at all. A bug in one customer's integration starts sending 50,000 requests/minute to `/search` (500x its quota), and latency degrades for everyone else. **Constraints** * 20 stateless API instances behind a round-robin load balancer, no sticky sessions * A Redis cluster already exists (used for caching elsewhere) * The limit must hold globally per customer. An instance-local counter lets one customer reach 100 × 20 = 2,000 req/min * Under 5ms added to the request path at p99 * Rolling reset: no bursting 100 requests either side of a minute boundary * Rate limiting must never be what takes the API down **What you observe** * The flood spreads evenly across all 20 instances, about 2,500/min each, and every instance sees a slice that looks locally plausible * p99 latency for other customers rises in the same window as the flood * A read-then-increment limiter passes every single-instance test, but quota is still exceeded once two instances overlap * The limiter's Redis call is \~0.3-1ms p99 when healthy, but p99 in the seconds when Redis is degraded * At full quota across all customers, the limiter makes \~83,000 calls/sec **The question:** Design the limiter end to end. Where does the counter live? What makes check-and-increment indivisible? What happens when Redis is down? What changes when limits are per-endpoint? I'd love to hear how you'd explain your design in a review, and especially what you'd do when Redis is slow or down, and why.
document fetched 2026-10-03 00:25 · canonical https://reddit.com/r/developersIndia/comments/1wvkscx/one_customer_sends_50000_reqmin_against_a_100min