Databricks System Design Interview: Distributed Job Scheduler

The interviewer opens

Hi, and thanks for joining. Welcome to your interview for the software engineer role at Databricks. We'll design a system together, with you sketching on the whiteboard on the right as you talk, and if you need to step out or take a break, the Pause button at the top right will stop things at any time.

The problem is a distributed job scheduler, which is close to the kind of orchestration work Databricks does. It's up in the Problem tab, so please read through the requirements and ask me anything you'd like clarified before you start drawing.

The problem

Design a system where users submit jobs to run immediately or on a recurring schedule, and the system runs them across a large fleet of workers, retrying failures so that no job is lost or run twice.

Must do

• Submit a job to run now or on a recurring schedule
• Run each job on one worker from a shared fleet
• Retry failed jobs according to the job's retry policy
• Let users see each run's status and history, and cancel a job

Must hold

• An accepted job is never lost, even when workers or servers die
• A job run's results must never be committed twice