System Design Interview
Design a production-grade API Gateway that provides authentication, authorization, rate limiting, request routing, and observability for multiple backend services.
Understand the basics of an API gateway: TLS termination, authentication, and routing to internal services.
What was missing: Clients called each backend service directly, so TLS, authentication and routing logic were duplicated in every service and inconsistently enforced.
Why that's risky: Only ~8% of requests reach the auth service. If the cache cluster is lost, auth load jumps 12x instantly; the auth tier needs surge headroom or the gateway must fail closed.
What gets added: An edge load balancer (TLS + WAF) in front of a 10-replica stateless gateway fleet, plus an auth/policy cache so token validation does not hit the auth service on every request.
Trade-offs: The gateway becomes a shared dependency and a place logic tends to accumulate. Keep it to cross-cutting concerns only — routing, authn/authz, limits, observability — and never business logic.
Implement traffic control features like per-tenant quotas and rate limits to protect backend services, and design a safe rollout strategy for gateway config changes.
What was missing: Any single tenant could saturate the gateway and starve everyone else, and every routing or policy change required a redeploy with no safe rollback.
Why that's risky: Rate-limit counters are the classic hot shard. Partitioning by tenant+api-key avoids one noisy tenant serialising the whole limiter.
What gets added: A distributed token-bucket rate limiter sharded by tenant/api-key ahead of the gateway, a versioned route/policy config store, and a canary deployment of the Order Service taking 10% of order traffic.
Trade-offs: Rate limiting adds ~2.5ms and a shared-state dependency. It fails open so a Redis shard loss degrades fairness rather than availability — the right trade for an availability-first edge, but it means limits are best-effort during incidents.
Implement rate limiting, observability, and asynchronous logging to ensure the gateway can handle production traffic safely and efficiently.
What was missing: There was no way to debug a single request end-to-end, prove SLO compliance, or retain access logs — and doing any of that synchronously would have blown the latency budget.
Why that's risky: If the log pipeline were synchronous and critical, a slow analytics store would add latency to every API call. Marking these edges async and non-critical is what keeps that blast radius contained.
What gets added: A fire-and-forget access-log pipeline (Kafka buffer → 20 ingester workers → analytics store) and an OTLP metrics/tracing collector fed by 5% head-based sampling.
Trade-offs: Async logging means logs can lag or, in a total buffer loss, be dropped. That is acceptable for observability data and unacceptable for anything billing-related, which must stay synchronous.
JWT validation is fast and avoids an auth dependency in the hot path (good for p99). Introspection enables immediate revocation and central control but adds latency and creates a dependency. Many production systems use JWT validation with short token TTLs plus a revocation mechanism for high-risk cases.
Scope limits correctly (per-tenant/user/key), keep checks O(1), and avoid a single centralized counter. Common approaches: sharded counters in Redis, local leaky-bucket with periodic sync, or tiered limiting (edge coarse limit + gateway fine limit).
Both, but with clear budgets. The gateway should enforce an overall deadline and avoid unbounded retries. Services often need their own timeouts/circuit breakers for downstream dependencies. The key is preventing retry storms and respecting a per-request latency budget.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
