System Design Interview
Inline AI code completion inspired by GitHub Copilot and Cursor: debounced requests, a gateway that cancels superseded requests, a small fast code model with continuous batching, repository context from an index of symbols and code snippets, a completion cache, routing between a fast and a larger model, and acceptance telemetry. A teaching design.
See why a general-purpose model called on every keystroke can't feel instant.
What was missing: Nothing yet.
Why that's risky: A suggestion that arrives after a second is useless: the developer has already typed past it. Most requests are for text that is outdated before they finish, and each one pays for a large model.
What gets added: One server that sends the whole current file to a large hosted model on every keystroke.
Trade-offs: The best model quality, at a latency and cost that make it unusable inline.
Make suggestions arrive before the developer types the next word.
What was missing: Every keystroke went to a slow, large model.
Why that's risky: The model only sees the open file and tabs, so it invents function names that don't exist in the repository.
What gets added: Editors wait for a short pause in typing before asking (debounce). A completion gateway cancels any request superseded by a newer one from the same editor, and a small, fast code model on our own GPUs, with continuous batching, produces suggestions in under 100ms.
Trade-offs: A small model is less capable than a large one, but at this latency it gets used, which matters more.
Suggest code that uses the repository's real functions and types.
What was missing: The model only saw open files and guessed at everything else.
Why that's risky: Identical prompts, common in boilerplate, are recomputed every time, and multi-line suggestions from the small model are weak.
What gets added: Changed files are queued and indexed: an indexer parses each file, extracts its symbols (functions, classes, imports) and embeds code snippets. The gateway adds the most relevant symbols and snippets to the prompt when the open files don't explain the code around the cursor.
Trade-offs: Retrieval adds a few milliseconds and some prompt length, in exchange for suggestions that compile.
Skip repeat work, and use a bigger model only where it pays off.
What was missing: Identical prompts were recomputed, and one model served every kind of request.
Why that's risky: Nobody knows which model or prompt change actually helps, one load balancer, and no per-seat limits.
What gets added: A cache of recent suggestions keyed by a hash of the prompt, and routing: single-line requests go to the fast model, multi-line and whole-function requests to a larger model.
Trade-offs: The larger model is slower, but multi-line requests are fewer and developers will wait a little longer for them.
Measure what developers keep, and survive the Monday-morning peak.
What was missing: No measure of whether suggestions were kept, one load balancer, and no per-seat limits.
Why that's risky: Remaining risk is GPU capacity: a sudden jump in seats needs GPUs that take time to add, so the larger model is degraded first.
What gets added: A sample of shown, accepted and rejected suggestions goes onto a queue and is aggregated per model, language and repository, which is how model and prompt changes are judged. A rate limiter per seat, a second load balancer, a replicated cache, more GPU servers, and monitoring of latency, cancellation rate and acceptance rate.
Trade-offs: Telemetry is sampled to keep its cost down, so small changes need a few days of data to judge.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
