Design a job scheduler: A Sample System Design Solution

The system design question

Design a job scheduler. Clients submit jobs to run at a given time or on a repeating schedule, and the system runs them reliably.

Agree on the scope (5 minutes)

Say up front what you're covering.

We'll design	We'll skip
Submitting a one-off job or a repeating one	What the jobs actually do
Finding jobs that are due and running them on time	Jobs that depend on other jobs
Retrying failed runs and recovering from crashed workers	Billing and quotas per client
Seeing the history of each job's runs	A visual dashboard

Then agree on what "good" means:

On time: a job starts within a few seconds of when it's due.
Runs once: each scheduled run happens, and its effect happens once, even when a worker dies halfway.
Never lost: once a client gets an OK back, the job is saved and will run.
Scales with the clock: lots of jobs land on the same minute, like the top of the hour, and that's fine.

Rough numbers (5 minutes)

These are assumptions, but reasonable ones: 10 million job definitions, 100 million runs a day, and a lot of them set for round times like midnight or the top of the hour, so peaks are about 20 times the average.

~1.2K
runs per second, on average: ~25K
runs per second at the top of the hour: ~10 GB
of job definitions: ~600 GB
of run history, kept for 30 days

Question	Rough math	What it means
How many runs a second?	100 million ÷ 86,400	About 1.2K a second on average, which isn't much
How bad is the top of the hour?	1.2K × 20	About 25K runs all due in the same few seconds. That's the real load
How big are the definitions?	10 million × about 1 KB	About 10 GB. Small
How much history?	100 million × about 200 bytes × 30 days	About 600 GB, so old runs need to be easy to drop

The overall design (10 minutes)

Each job gets one row per upcoming run. Pollers find rows that are due, claim them, and queue them. Workers run them and mark them done.

A client submits a job: what to call, when to run it (a time, or a repeating schedule like a cron string), and a retry policy. We save it and create a row for its next run.
The runs table is split by run time and a shard number, so the rows due in any given minute are spread across a known set of partitions.
A small group of pollers, each owning some shards, reads its slice of due runs every second, claims them and puts them on a queue.
Workers take runs from the queue, run them, and report success or failure. Failures are retried with growing gaps until the retry limit.
When a repeating job's run is claimed, the next run's row is created, so the schedule keeps going even if this run is slow.

What the apps need to do

Action	Who does it	What's sent
Create a job	Client	Target to call, schedule or run time, payload, retry policy, request key
Pause, resume or delete a job	Client	Job ID
List a job's runs	Client	Job ID, time range, page cursor
Report a run's result	Worker	Run ID, attempt number, success or failure, output

What we store and where

Data	Where it goes	Why
Jobs	Database, split by job ID	Looked up by ID. Small and rarely changes
Runs	Database, split by time bucket plus shard number	Pollers read 'what's due now', which needs to be one small, spread-out slice
Run history	Same runs table, old buckets dropped after 30 days	Dropping a whole old time bucket is much cheaper than deleting rows one by one
Due-run queue	Message queue	Smooths the top-of-the-hour pile and spreads it over workers

Step 1: Tables, indexes and partition keys (12 minutes)

Two tables. Jobs holds the definition: job ID, owner, target, payload, schedule, retry policy, and whether it's paused. Its partition key is the job ID, because you always look a job up by ID.

Runs holds one row per planned run: job ID, scheduled time, status (waiting, claimed, done, failed), attempt number, who claimed it, and a lease end time. The job ID plus scheduled time is unique, so a run can't be created twice for the same slot. That's idempotency built into the key.

The question that matters is how pollers find due runs. If the runs table were split by job ID, 'what's due right now' would have to ask every partition. So split it by time instead: the partition key is the minute the run is due. Pollers then read exactly one partition per minute.

That creates a hot spot, though. Every run due at midnight lands in the same partition. So add a shard number, 0 to, say, 63, picked from a hash of the job ID. The partition key becomes the minute plus the shard. Midnight's 25K runs spread over 64 partitions, and each poller owns a few shard numbers. That's the sharding trade: one small extra field fixes the hot spot.

Inside each partition, sort by scheduled time, then status, so a poller reads 'waiting runs up to now' as one range.
Add a second index on job ID and scheduled time, so 'show me this job's history' doesn't scan every bucket.
Old time buckets can be dropped whole after 30 days.

Step 2: Running each job once when workers crash (10 minutes)

A poller claims a run with a conditional update: 'set status to claimed and lease end to now plus 60 seconds, only if it's still waiting'. If two pollers ever overlap, only one wins. Then it puts the run on the queue.

A worker that's running a long job keeps pushing the lease end forward every few seconds. If the worker dies, the lease runs out, and the poller treats the run as waiting again. That's how a crashed run gets picked up.

This means a run can occasionally start twice: say a worker was frozen, not dead, and comes back after its lease ran out. You can't fully stop that, so the honest promise is 'at least once', plus a run ID. Pass the run ID and attempt to the job's target, so it can skip work it already did. When a worker reports back, it only wins if its attempt number still matches.

Retries

On failure, bump the attempt number and set the run back to waiting with a later scheduled time: 10 seconds, then a minute, then 10 minutes. After the last try, mark it failed and tell the owner. Moving the time pushes the retry into a later bucket, so it doesn't clog the current one.

Step 3: Repeating jobs and the top-of-the-hour spike (8 minutes)

For a repeating job, only keep the next run in the table, not all future ones. When a run is claimed, work out the next time from the schedule and insert that row. The unique key on job ID plus time means a retried insert is harmless.

Create the next run when this one is claimed, not when it finishes. Otherwise a job that runs for 10 minutes would push its own schedule back every time.

For the spike, pollers can read a few seconds ahead and hold runs until they're due, so the queue fills smoothly instead of all at once. The queue then absorbs the rest. Clients that don't need exact timing can also opt in to a small random delay, which flattens the spike even more.

Choices you should be able to defend

We chose	Instead of	Because	We'd reconsider if
Partition runs by minute plus shard	Partition by job ID	Finding due runs reads a few small partitions, and midnight is spread out	We mostly looked runs up by job, not by time
Leases that expire	A worker holding a run until it says it's done	A dead worker's runs come back on their own	Jobs were so long that keeping a lease alive was hard
At least once, plus a run ID	Promising exactly once	Exactly once across a crash isn't possible on our side. The target can skip repeats	Never, but we'd help targets that can't dedupe
Only the next run stored	Every future run stored up front	Far fewer rows, and changing a schedule only touches one row	Clients needed to see far-off future runs

What could go wrong (5 minutes)

What happens	What we do about it
A worker dies mid-run	The lease runs out, the run goes back to waiting, and another worker picks it up
A poller dies	Its shards are given to another poller, which reads from the oldest waiting run, so nothing's skipped
The queue backs up	Runs start late. Add workers, and alert if 'due but not started' grows
The database is slow at midnight	Pollers read ahead a few seconds, and more shards spread the load
A job keeps failing	Retries back off and stop at the limit, so one bad job can't hog workers

Follow-up questions (5 minutes)

How would you support jobs that depend on other jobs?

Store the dependencies as edges, and only create a run row for a job once all the runs it depends on are done.

How would you stop one client hogging the workers?

Give each client a limit on runs in flight, and rate limit how fast their runs are queued. Their extra runs wait in the table.

What about time zones and daylight saving?

Store the schedule with its time zone and work out each next run in that zone, then save the run time in UTC.

How the 60 minutes add up

Minutes	What you're doing
0–5	Agree on what you're building
5–10	Rough numbers
10–20	Draw the overall design
20–32	Tables, indexes and partition keys
32–42	Running each job once when workers crash
42–50	Repeating jobs and the top-of-the-hour spike
50–55	What could go wrong
55–60	Follow-up questions

System design: design a job scheduler