The system design question
Design a system where users submit jobs to run now or on a schedule. It runs them across lots of workers and handles retries and failures, so jobs don't get lost or run twice.
Agree on the scope (5 minutes)
Say up front what you're covering.
| We'll design | We'll skip |
|---|---|
| Submitting a job to run now, at a set time, or on a repeating schedule | What the jobs actually do |
| Handing due jobs to workers | Packaging and shipping job code |
| Noticing when a worker dies and running the job again | Jobs that depend on other jobs |
| Retries, and stopping a job from running twice | Billing |
Then agree on what "good" means:
- Never lost: once we accept a job, it runs, or it ends up clearly marked as failed.
- Not run twice: a job's effects happen once, even when workers crash or the network splits.
- On time: a scheduled job starts within a few seconds of when it's due.
- Scales out: more jobs or more workers just means adding machines.
Rough numbers (5 minutes)
These are assumptions, made up but reasonable: 100 million job runs a day, a mix of one-off and repeating, 10,000 workers, and peaks about 10 times the average, since lots of schedules land right on the hour.
- ~1.2K
- job runs started a second, on average
- ~12K
- a second at a peak
- 10K
- workers
- ~20 TB
- of run history a year
| Question | Rough math | What it means |
|---|---|---|
| How many starts? | 100 million ÷ 86,400 seconds, then × 10 | About 12K a second at the top of the hour. Too much for one database row lock at a time |
| How many heartbeats? | 10,000 workers, one every 5 seconds | About 2K a second. Easy, if they don't all hit one table |
| How much history? | 100 million runs × about 500 bytes × 365 days | About 20 TB a year, so old runs move to cheap storage |
The overall design (10 minutes)
- A user submits a job with a request ID: what to run, when, how many retries and a timeout. We save it and return a job ID.
- Repeating jobs are saved once with their schedule. The scheduler creates a run row for each time the job is due.
- Schedulers each own part of the jobs, find runs that are due, and push them onto the run queue.
- A worker takes a run, gets a lease on it for a short time, and keeps renewing the lease with heartbeats while it works.
- When the worker finishes, it marks the run done. If the lease runs out first, the run goes back on the queue for another worker.
What the apps need to do
| Action | Who does it | What's sent |
|---|---|---|
| Submit a job | User or another service | What to run, when or a cron schedule, retries, timeout, request ID |
| Get a job and its runs | User | Job ID. Returns its state and recent runs |
| Cancel or pause a job | User | Job ID |
| Take a run | Worker | Worker ID, what kinds of jobs it can run. Returns a run and a lease |
| Heartbeat | Worker | Run ID, lease token |
| Finish a run | Worker | Run ID, lease token, result or error |
What we store and where
| Data | Where it goes | Why |
|---|---|---|
| Jobs | Relational database, split by job ID | What to run, the schedule, retry policy, owner, and paused or not |
| Runs | Same database, same split | One row per attempt to run a job at a time: due time, state, attempt number, lease owner and lease expiry |
| Run queue | Message queue | Runs that are due and ready for a worker |
| Request IDs | Same database | So submitting the same job twice returns the first one |
| Old runs | Object storage | Finished runs older than a few weeks, kept for history |
Step 1: Getting due jobs to workers (10 minutes)
Store each run with its due time and index on (state, due time). A scheduler loop asks 'which waiting runs are due now?' every second, flips them to 'queued' and pushes them onto the queue. Flip the state and push in a way that survives a crash: write an outgoing row in the same transaction and publish from it, so a run can't be marked queued but never sent.
One scheduler can't handle 12K runs a second at the top of the hour. Split jobs by job ID across database machines with sharding, and give each scheduler a set of shards. If a scheduler dies, another one takes over its shards. A short lease on each shard makes sure only one scheduler owns it at a time.
For repeating jobs, when a run is queued, also create the next run with its next due time. Then there's always exactly one future run per schedule, and you never need to scan every job to see what's due.
Pull, not push
Workers pull runs from the queue when they're free. You don't have to track every worker's load, and a slow worker just takes fewer runs. Separate queues per job type or priority keep big batch jobs from starving quick ones.
Step 2: Leases, heartbeats and dead workers (12 minutes)
When a worker takes a run, it gets a lease: 'this is yours for 30 seconds'. The run row records the worker, a lease token and the expiry. The worker heartbeats every 10 seconds or so to push the expiry out. If the worker dies, heartbeats stop, the lease runs out, and a sweeper puts the run back to 'waiting' with the attempt number plus 1.
The tricky case is a worker that isn't dead, just stuck for a while, like a long garbage collection pause. Its lease runs out, another worker takes the run, and then the first one wakes up and keeps going. Now two workers are running the same job. You can't stop that from happening, only stop it from doing damage.
Use a fencing token: every new lease gets a higher number. The worker sends it with every heartbeat and with 'finish'. The database only accepts an update if the token matches the current lease, so the old worker's 'done' is rejected. If the job writes to other systems, pass the token along so they can reject stale writes too. This is a consistency problem, so the lease itself has to live somewhere with strong consistency, like the jobs database.
Step 3: Retries without running twice (8 minutes)
Be honest in the interview: with workers that can crash after doing the work but before saying so, you can't promise a job runs exactly once. You can promise it runs at least once, and make running it again harmless. That's what 'doesn't run twice' really means in practice.
Give each run a stable ID, and have the job use it as an idempotency key for anything it changes. If it writes a file, write to a name based on the run ID. If it calls a payment API, pass the run ID as the request key. A retry then finds the work already done.
Retries back off: wait 10 seconds, then 40, then a few minutes, with a bit of randomness so a broken dependency isn't hammered by every job at once. After the max attempts, mark the run failed and park it in a dead-letter list for a person to look at. A timeout counts as a failure too, so a job that hangs forever doesn't hold a worker forever.
Choices you should be able to defend
| We chose | Instead of | Because | We'd reconsider if |
|---|---|---|---|
| Jobs and leases in a relational database | Only a message queue | Leases, retries and fencing need conditional updates on one row, and history needs queries | Jobs were all short and simple enough that the queue's own redelivery was enough |
| At least once plus idempotent jobs | Promise exactly once | Exactly once isn't possible when workers can crash mid-job, so we make repeats safe instead | Never. Be upfront about it |
| Workers pull runs | Scheduler pushes to chosen workers | Simple, and load balances itself | Jobs needed special hardware and placement had to be planned carefully |
| One future run per schedule | Scan all schedules every tick | Due jobs come straight off an index | There were only a few thousand schedules, where a scan is simpler |
What could go wrong (5 minutes)
| What happens | What we do about it |
|---|---|
| A worker dies mid-job | Heartbeats stop, the lease runs out, and the run goes back to waiting for another try |
| A worker is paused, not dead, and comes back | Its fencing token is old, so its heartbeat and 'finish' are rejected and it stops |
| A scheduler dies | Its shard leases run out and other schedulers take them. Due runs are a few seconds late, not lost |
| The queue loses or repeats a message | The run row is the source of truth. A sweeper re-queues runs stuck in 'queued', and a duplicate fails to get the lease |
| A job fails every time | Back off, give up after the max attempts, mark it failed and alert the owner |
Follow-up questions (5 minutes)
How would you support jobs that depend on other jobs?
Store the dependencies as a graph. A run only becomes due when every run it depends on has finished. Finishing a run checks its dependents.
What happens when a million jobs are all scheduled for midnight?
Spread them across shards and queues, and let users add a little jitter. Workers pull at their own pace, so the queue absorbs the spike and things finish a bit later.
How do you stop one user from hogging every worker?
Give each user or team a limit on running jobs, and pick the next run fairly across users rather than first come first served.
How the 60 minutes add up
| Minutes | What you're doing |
|---|---|
| 0–5 | Agree on what you're building |
| 5–10 | Rough numbers |
| 10–20 | Draw the overall design |
| 20–30 | Getting due jobs to workers |
| 30–42 | Leases, heartbeats and dead workers |
| 42–50 | Retries without running twice |
| 50–55 | What could go wrong |
| 55–60 | Follow-up questions |
Key concepts used here
- Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
- Idempotency and retriesMake sure a retried request doesn't happen twice.
- Consistency and the CAP theoremWhat happens when copies of your data can't agree.
- Sharding and consistent hashingSplit data across machines and keep it balanced.
- Relational vs non-relational databasesPick a database based on how you'll actually use it.
Further reading
We wrote this sample answer ourselves.
Practice Loop is independent and isn't affiliated with or endorsed by Databricks.