System Design Interview
Distributed job scheduler inspired by Airflow, Kubernetes CronJobs and the distributed cron described in Google's SRE book: jobs saved with their next fire time, partitioned schedulers holding leases and firing from in-memory timer wheels, exactly-once claims with a conditional insert, an executor fleet behind a dispatch queue, heartbeats, retries with backoff, dead letters, and a run status cache. A teaching design.
See why a single crontab box misses runs and can't grow.
What was missing: Nothing yet.
Why that's risky: If the server is down at 02:00, the 02:00 runs never happen. Long jobs pile up on one machine, and a burst at the top of the hour overloads it. Job code shares the box with the scheduler.
What gets added: One server that keeps jobs in a table and executes every due job itself, once a minute.
Trade-offs: This is how cron works on one machine. It's fine for a handful of jobs, not for a company's worth.
Separate deciding when a job is due from executing it.
What was missing: One machine decided when jobs were due and also executed them.
Why that's risky: There is one scheduler. Run two for safety and both fire every job, so jobs execute twice. Nothing retries a failed run.
What gets added: A job API that saves jobs with their next fire time, a scheduler that finds due jobs and puts each run on a dispatch queue, executor workers that call the job's target, and a runs table for history.
Trade-offs: A run starts a moment after its due time because it passes through a queue. Scheduled jobs almost never need sub-second precision.
Run several schedulers without ever firing a job twice.
What was missing: Two schedulers meant every job fired twice.
Why that's risky: A failed run is simply marked failed, and a run whose executor hangs looks like it's still going forever.
What gets added: The jobs table is split into partitions, and each scheduler holds a renewable lease on some of them. A scheduler loads the next few minutes of due jobs into an in-memory timer wheel, and claims each run with a conditional insert unique on (job, due time), so even an overlapping scheduler can't claim the same run.
Trade-offs: Leases take a few seconds to expire, so if a scheduler dies, its jobs can start a few seconds late.
Recover failed and stuck runs automatically, and tell owners about the ones that can't be recovered.
What was missing: Failed runs stayed failed, and a hung executor held its run forever.
Why that's risky: Dashboards and CI pipelines poll run status from the database every few seconds, and there is one load balancer.
What gets added: Executors send heartbeats while a run is going, and a run without one is treated as failed. Failed runs go to a retry queue, and a retry worker executes them again after an exponential backoff. Runs that use up their retries land on a dead-letter list, and the owner is alerted.
Trade-offs: Retries mean a job may run more than once after a partial failure, so job targets should be idempotent. The run id is passed to them for that.
Serve status polling cheaply, and spot schedule lag before owners do.
What was missing: Status polling read the database, one team could register millions of jobs, and nobody watched schedule lag.
Why that's risky: Remaining risk is low. A huge burst at the top of the hour still queues behind the executors, but no run is skipped.
What gets added: A run status cache written by executors, a rate limiter with per-team quotas, a second load balancer, a replicated runs database, more executors, and monitoring of schedule lag (due time against start time), failure rate and dead letters.
Trade-offs: Jobs due at :00 may start a few seconds late during the hourly burst. Spreading start times (jitter) is offered to owners who can accept it.
You're in the middle of an interview session. Leaving now will end your current attempt.
Explore concept overviews, real-system examples, key tradeoffs, and interview talking points for each roadmap section.
You've conquered this phase. These are the skills you now own:
