System Design Interview
Retrieval-augmented Q&A over company documents inspired by Perplexity, Notion AI and Glean: documents split into overlapping passages and embedded by workers, hybrid vector and keyword retrieval, cross-encoder re-ranking, answers grounded in cited passages, an answer cache keyed by meaning and document version, and access lists enforced inside retrieval. A teaching design.
See why keyword matching plus a giant prompt is slow, expensive and often wrong.
What was missing: Nothing yet.
Why that's risky: Keywords miss documents that say the same thing in other words, and match many that don't help. Long prompts are slow and costly, and they overflow the model's context window. The model may answer from its own memory instead of the documents.
What gets added: One server that finds documents containing the question's keywords and pastes all of them into the LLM prompt.
Trade-offs: No indexing work at all, paid for on every single question.
Send the model only the few passages that answer the question.
What was missing: Whole documents went into the prompt, found by keyword.
Why that's risky: Pure vector search misses exact terms like error codes and product names, 20 passages is still a lot of prompt, and the same questions are answered from scratch again and again.
What gets added: Uploads are saved and queued. Embedding workers split each document into overlapping passages of about 500 tokens, turn each into a vector, and store it in a vector index. A question is embedded the same way, its 20 nearest passages are retrieved, and only those go to the LLM, which must answer from them and cite them.
Trade-offs: Passages that are too small lose context, and passages that are too large dilute the match. About 500 tokens with some overlap is a common starting point.
Find exact terms as well as meanings, send fewer and better passages, and skip repeat work.
What was missing: Vector search alone missed exact terms, prompts carried 20 passages, and repeat questions cost the same as new ones.
Why that's risky: Everyone sees passages from every document, including ones they aren't allowed to open, and edited documents can leave old passages behind.
What gets added: Retrieval combines nearest-neighbour search with keyword search (hybrid), a small cross-encoder re-ranks the candidates and keeps the best 5, and an answer cache returns recent answers when the question means the same thing and the cited documents haven't changed.
Trade-offs: Re-ranking adds about 40ms but noticeably improves which passages reach the model. Cached answers can be slightly stale until a cited document changes.
Never show a passage the user can't open, and keep answers current as documents change.
What was missing: Retrieval ignored who may read what, and stale passages outlived edits.
Why that's risky: One load balancer, no per-user limits on an expensive LLM, and no measure of answer quality.
What gets added: Each passage carries its document's access list, and retrieval filters to documents the user can read before ranking. Changes from connected apps are re-processed: a new version's passages replace the old ones, and deleted documents are removed from the index.
Trade-offs: Access lists are copied into the index, so a permission change takes effect after the document is re-processed, usually within a minute.
Control LLM spend, and know when answers get worse.
What was missing: No spending limits, a single load balancer and index, and no idea whether answers were good.
Why that's risky: Remaining risk is the LLM provider: if it is down, answers stop, though retrieval can still return the relevant passages.
What gets added: A rate limiter per user and workspace, a second load balancer, index and re-ranker replicas, more query instances, and monitoring of latency, ingest lag, thumbs-down rate and how often the cited passages actually contain the answer.
Trade-offs: Measuring quality needs a sample of graded answers, which costs people's time or an extra model call.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
