System Design Interview
Neural machine translation for text and documents, inspired by DeepL and Google Translate: stateless APIs in front of a batched GPU model fleet, a translation memory that reuses every sentence already translated, glossaries per customer, and whole documents translated in the background. A teaching design.
See why translating inside one server runs out of GPU at once.
What was missing: Nothing yet.
Why that's risky: A single GPU translates one batch at a time; a 100-page document blocks everyone behind it.
What gets added: One server that sends every sentence to one GPU running the model and stores documents in the database.
Trade-offs: Easy to start, but the web tier and the GPU can't grow separately.
Scale the web tier and the model separately.
What was missing: One server did everything, and documents blocked text.
Why that's risky: Every sentence still goes to a GPU, even ones translated a thousand times before. Documents are still translated inside the upload request.
What gets added: Stateless API instances behind a load balancer, a fleet of GPU model servers that batch sentences from many requests, and file storage for documents.
Trade-offs: GPUs are the cost; batching raises throughput but adds a few milliseconds of waiting.
Never translate the same sentence twice.
What was missing: The GPUs re-translated sentences they had translated many times before.
Why that's risky: A cached sentence is translated without its neighbours, which matters for pronouns and gender; include the previous sentence in the key for languages where it changes the result.
What gets added: A translation memory: every translated segment is saved by its source text, language pair and glossary version, with the hottest ones in a cache. Only new sentences reach the model.
Trade-offs: Changing a glossary changes its version, so old entries stop matching rather than being purged.
Translate big files without holding a request open.
What was missing: Long documents were translated inside the upload request and timed out.
Why that's risky: A huge upload can flood the GPUs; give document segments a lower priority than live text.
What gets added: The upload only saves the file and a job; a queue feeds document workers that split the file into segments, translate them through the same memory and model, rebuild the layout and mark the job done.
Trade-offs: Users wait a minute for a document instead of a spinner that fails.
Protect the GPUs and remove single points of failure.
What was missing: One API key could saturate the GPUs, and the load balancer and databases were single points of failure.
Why that's risky: Free users hit limits sooner during surges.
What gets added: A rate limiter on characters per key, a second load balancer, replicas for both databases, a bigger GPU fleet for surges, and monitoring of GPU queue time.
Trade-offs: Paying for headroom on GPUs is expensive; scale them on queue time, not CPU.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
