System Design Interview
API gateway platform inspired by Envoy/Kong-style gateways: local JWT validation with cached signing keys, a control plane pushing versioned routes, weighted canary routing, distributed token-bucket rate limiting, circuit breakers and async access logs. A teaching design.
See why authenticating every call against a database doesn't fit a 50 ms budget.
What was missing: Nothing yet.
Why that's risky: Every API call becomes a database query before any real work happens, and one proxy carries 100K calls per second.
What gets added: One proxy that looks up each caller's token in the identity database and forwards the request.
Trade-offs: Revocation is instant, but the identity database is on every request's critical path.
Authenticate without a database call per request.
What was missing: Every call queried the identity database.
Why that's risky: Routes live in each gateway's config file, so a change means redeploying the fleet, and new backend versions can't be tried safely.
What gets added: An edge load balancer with TLS and WAF, a stateless gateway fleet that validates JWTs locally with the auth service's public signing keys (JWKS) and looks up API keys in a cache, and an auth service that issues tokens, rotates keys and answers revocation checks.
Trade-offs: A locally validated token stays valid until it expires unless the revocation list catches it, so tokens are short-lived (minutes).
Change routing safely without redeploying gateways.
What was missing: Routing was baked into gateway config files.
Why that's risky: One client can still send unlimited requests and starve everyone else.
What gets added: A control plane that stores versioned routes, policies and quotas and pushes them to every gateway, where they are swapped atomically in memory; weighted routing sends a small share of traffic to a canary version of a backend.
Trade-offs: Pushed config is eventually consistent: for a few seconds some gateways may run the previous version.
Enforce per-tenant quotas across the whole fleet.
What was missing: No quotas: one tenant could starve all others.
Why that's risky: Gateways write access logs synchronously, and nobody can see per-route errors or trace a slow request.
What gets added: A distributed rate limiter with token buckets per tenant and API key in Redis; each gateway takes a small local allowance so most requests skip the round trip.
Trade-offs: Local allowances allow a tenant to slightly exceed its quota for a moment, in exchange for not paying a Redis call per request.
Record every call without slowing it, and see and contain failures.
What was missing: Logging was synchronous and failures were invisible.
Why that's risky: Remaining risk is regional: the fleet runs in several regions behind global DNS or anycast.
What gets added: Gateways write access log entries and trace spans to a buffer without waiting; ingesters batch them into object storage and compute per-route metrics. Circuit breakers stop forwarding to a backend that keeps failing, and dashboards alert on latency, errors and breaker trips.
Trade-offs: Logs arrive seconds after the call; under extreme load the buffer may sample rather than drop requests.
JWT validation is fast and avoids an auth dependency in the hot path (good for p99). Introspection enables immediate revocation and central control but adds latency and creates a dependency. Many production systems use JWT validation with short token TTLs plus a revocation mechanism for high-risk cases.
Scope limits correctly (per-tenant/user/key), keep checks O(1), and avoid a single centralized counter. Common approaches: sharded counters in Redis, local leaky-bucket with periodic sync, or tiered limiting (edge coarse limit + gateway fine limit).
Both, but with clear budgets. The gateway should enforce an overall deadline and avoid unbounded retries. Services often need their own timeouts/circuit breakers for downstream dependencies. The key is preventing retry storms and respecting a per-request latency budget.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
