System Design Interview
One API in front of many model providers, inspired by OpenRouter and Portkey: API keys checked from a cache, token usage metered in the background, requests routed to a healthy provider with automatic fallback, and per-key rate limits and budgets. A teaching design.
See why a gateway that does everything in one place can't keep up.
What was missing: Nothing yet.
Why that's risky: Every request does two database round trips, and one server carries all of it. If the provider is down, every app is down.
What gets added: One server that looks up the API key in the database, calls one provider, and writes a usage row before replying.
Trade-offs: Simple to build, but the gateway is now the weakest link in every customer's app.
Keep key checks and usage writes off the hot path.
What was missing: Every request hit the database twice, and one server held everything.
Why that's risky: Still one provider: when it fails, so does every request.
What gets added: Stateless gateway instances behind a load balancer, a cache of API keys, and a queue that hands token counts to metering workers, which write usage in batches.
Trade-offs: Usage appears in dashboards a few seconds late; a revoked key keeps working until its cache entry expires (keep that TTL short).
Pick a healthy provider for each request and fail over automatically.
What was missing: One provider outage took down every app behind the gateway.
Why that's risky: Retries can double the cost of a request; only retry before any tokens have been streamed back.
What gets added: A router that tracks each provider's error rate and latency, sends each request to the best healthy one, and retries a failed call once on another provider.
Trade-offs: Two providers cost more to integrate, and their answers differ slightly for the same model family.
Stop one key from starving everyone else or overspending.
What was missing: A runaway loop in one app could use up the providers' capacity and run up a huge bill.
Why that's risky: Spend lags by a few seconds, so a key can go slightly over its budget.
What gets added: A rate limiter with token buckets per key, and a spend total kept beside each key in the cache, updated by the metering workers, so the gateway can refuse keys that are over budget.
Trade-offs: Hard limits protect everyone but can surprise a customer mid-launch; send a warning at 80% of budget.
Remove the single points of failure and see provider trouble first.
What was missing: The load balancer and both databases were single points of failure, and nobody saw a provider slowing down.
Why that's risky: Failover still drops requests that were in flight.
What gets added: A second load balancer next to the first (apps reach either one), standby replicas for the keys and usage databases, more gateway and router instances, and monitoring per provider.
Trade-offs: Replicas double the database bill for safety you hope not to need.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
