System Design Interview
AI chat assistant inspired by ChatGPT and Claude: a stateless chat API that streams replies token by token, a GPU pool with continuous batching, prefix-aware routing to reuse KV caches, a context cache, per-plan token quotas metered from a queue, prompt moderation, and a smaller model for overflow. A teaching design.
See why GPU inference can't live inside the web server.
What was missing: Nothing yet.
Why that's risky: A reply takes seconds to generate, and the user stares at a blank screen the whole time. The GPU serves one request at a time, so a few users fill it. Web traffic and GPU work can't be scaled separately.
What gets added: One server that saves the prompt, runs the model on its own GPU, and replies when the whole answer is generated.
Trade-offs: Simple, but expensive GPU time is wasted between requests and the web tier can't scale without buying GPUs.
Scale cheap web servers and expensive GPUs separately, and show the reply as it is generated.
What was missing: The web server owned the GPU, and the user waited for the whole reply.
Why that's risky: Every request rebuilds the context from the database, and requests land on any GPU, so the model reprocesses the whole conversation each turn.
What gets added: A stateless chat API that saves the prompt and streams the reply token by token over server-sent events, a pool of GPU servers that batch many requests together (continuous batching), a conversations store partitioned by conversation id, and a history service.
Trade-offs: Streaming holds a connection open for the length of the reply. That's cheap on the chat API, and it makes the wait feel much shorter.
Stop paying to reprocess the same conversation every turn.
What was missing: Each turn reread the conversation from the database and landed on a random GPU, which recomputed the whole conversation's attention state.
Why that's risky: Nothing limits how much one user can send, nothing checks prompts, and usage isn't measured.
What gets added: A context cache holding the last few turns of active conversations, and an inference router that sends a conversation back to the GPU that already holds its prefix in its KV cache, or else to the least loaded one.
Trade-offs: Prefix routing can make one GPU busier than the rest. The router falls back to the least loaded GPU when the preferred one is full.
Make GPU time fair and paid for, and keep disallowed prompts off the GPUs.
What was missing: No limits per user, no usage numbers for billing, and no check on what was sent to the model.
Why that's risky: When every GPU is busy, requests queue at the router until they time out. There is no cheaper fallback, and one load balancer.
What gets added: A rate limiter enforcing each plan's messages per hour and token budget, a moderation classifier in front of the GPUs, and usage events that a metering worker adds up per user per day into a usage store that quotas read from.
Trade-offs: Metering is asynchronous, so a user can slightly overshoot their quota before it catches up.
Keep answering during a launch-day surge without buying GPUs for the peak.
What was missing: No fallback when the GPUs were full, one load balancer, and no view of time to first token.
Why that's risky: Remaining risk is GPU supply: the fleet can't grow in minutes, so a large surge is handled by degrading, not scaling.
What gets added: A pool running a smaller, faster model that takes simple prompts and overflow when the main pool is saturated, a rate limiter that sheds free-tier load first, a second load balancer, a replicated conversations store, more GPU servers, and monitoring of time to first token, tokens per second and GPU utilisation.
Trade-offs: Under load, some users get the smaller model's answers. Many products show this as "using a faster model right now".
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
