System Design Interview
Distributed rate limiting service inspired by Envoy's global rate limit service and similar designs: central atomic counters in sharded Redis with sliding windows, leased local allowances for hot keys, fail-open behaviour, throttle analytics and cross-region count sync. A teaching design.
See why per-server limits don't add up to a real limit.
What was missing: Nothing yet.
Why that's risky: With 80 servers a client effectively gets up to 80 times its limit, the load balancer decides which bucket it hits, and every restart resets the counts.
What gets added: Each app server keeps an in-memory token bucket per client and rejects requests over its limit.
Trade-offs: Zero latency and no dependencies, but the limit isn't a limit.
Enforce one limit per client across the whole fleet, without boundary bursts.
What was missing: Counters were per server.
Why that's risky: Every request now costs a Redis round trip, and one very hot key lands on one Redis shard.
What gets added: An edge proxy that asks a rate limit service before forwarding; the service matches rules (cached from the rules DB) and checks and increments counters in a sharded Redis with one atomic script, using a sliding window so a burst at a window boundary can't double the limit.
Trade-offs: A sliding-window counter approximates the true window by weighting the previous one; it's cheap and accurate enough.
Cut Redis round trips for the busiest clients.
What was missing: Every check went to Redis, and hot keys overloaded their shard.
Why that's risky: Nobody can see who is being throttled, limits are per region only, and losing Redis loses all counts.
What gets added: Each limiter node leases a small batch of tokens from Redis per hot key and spends them locally, going back to Redis only when the batch runs out.
Trade-offs: A client can slightly exceed its limit (by up to one batch per node) in exchange for far fewer Redis calls.
See who is throttled, approximate global limits across regions, and recover counters after failures.
What was missing: No visibility, limits only per region, and no recovery for counters.
Why that's risky: Remaining risk is policy: wrong rules can throttle good customers, so changes are reviewed and rolled out gradually.
What gets added: A stream of allowed and throttled decisions feeding throttle analytics, a cross-region sync worker that exchanges counts every second for approximate global limits, periodic counter snapshots for recovery, and monitoring of latency, deny rate, Redis health and fail-open events.
Trade-offs: Global limits lag by about a second across regions.
Shard counters by user hash and use sliding windows with probabilistic sampling.
Use async queues to reconcile counter deltas and accept minor drift with periodic sync.
Edge placement blocks abusive traffic early and reduces downstream load.
The queue smooths counter updates and lets workers reconcile state without blocking requests.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
