System Design Interview
Platform for long-running AI agents inspired by OpenAI's Assistants and Claude's agent tools: tasks accepted and queued, the agent loop in workers that save every step before the next, ownership through conditional updates and heartbeats so a crashed run resumes, step, token and time budgets, a tool gateway with per-task permissions, code in throwaway microVMs, approvals for risky actions, and live progress over open connections. A teaching design.
See why a loop that runs for minutes can't live inside an HTTP request.
What was missing: Nothing yet.
Why that's risky: Agent tasks take minutes and dozens of steps. Connections time out, a deploy or crash loses all progress, the user sees nothing until the end, and a looping agent can run up an unbounded bill.
What gets added: One server that runs the agent loop inside the request: ask the LLM for a step, call a tool, repeat, and reply when the task is done.
Trade-offs: Fine for a two-step demo, not for real tasks.
Accept a task instantly and run the loop in the background.
What was missing: The loop ran inside the request.
Why that's risky: Progress lives only in the worker's memory, so a crash or deploy restarts the task from scratch, possibly repeating actions. Nothing caps a runaway loop, and two workers can pick up the same task.
What gets added: A task API that saves each task and queues it, agent workers that run the loop and call tools through a tool gateway, and a progress service the client polls.
Trade-offs: The client polls for progress instead of waiting, which is the right shape for long work.
Resume a crashed run where it stopped, never run a task twice at once, and stop runaway loops.
What was missing: Progress was lost on a crash, two workers could drive one task, and nothing stopped a loop.
Why that's risky: Tools can do anything the agent asks, including running code on shared machines and taking irreversible actions.
What gets added: Every step (the model's decision, the tool used, the result) is saved to a step log before the next step starts. A worker takes ownership of a task with a conditional update and keeps it with heartbeats; if the heartbeats stop, another worker resumes from the last saved step. Each task has a step, token and time budget.
Trade-offs: Saving every step costs a write per step, which is small next to an LLM call.
Let agents run code and take actions without putting the platform or the user at risk.
What was missing: Tools ran with broad access and nothing stopped irreversible actions.
Why that's risky: Users still poll for progress, one load balancer, and no per-account cap on running tasks or LLM spend.
What gets added: Generated code runs in throwaway microVMs with no network beyond approved hosts and fixed limits. Every tool call uses the task's own permissions, and actions marked risky (sending email, payments, deleting data) pause the task until the user approves.
Trade-offs: Approvals slow tasks down, but an agent that sends the wrong email can't take it back.
Show each step as it happens, contain cost per account, and catch stuck or failing agents.
What was missing: Clients polled for progress, one account could start thousands of tasks, and nobody saw stuck or failing agents.
Why that's risky: Remaining risk is the LLM provider: if it is slow or down, tasks pause, though none are lost.
What gets added: Each step goes onto a progress queue and is pushed to watching clients over open connections, which catch up from the step log on reconnect. A rate limiter caps running tasks and LLM tokens per account. A second load balancer, a replicated tasks database, more workers, and monitoring of task success rate, tool errors, cost per task and tasks with no heartbeat.
Trade-offs: Live updates hold a connection per watching client, a small cost for showing every step.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
