Databricks System Design Interview: Multi-Threaded Logging to a Single File

The interviewer opens

Hi, and thanks for joining. Welcome to your interview for the software engineer role at Databricks. We'll design a system together, and you can sketch on the whiteboard on the right of your screen as you talk. If you need to step out or take a break, the Pause button at the top right will pause the interview.

The problem is a multi-threaded, in-process logger, the kind of low-level building block that data platforms lean on heavily. It's up in the Problem tab, so please read through the requirements and ask me anything you'd like clarified before you start drawing.

The problem

Design an in-process logging library that many threads in one program call concurrently, writing every entry to a single local file complete and in the order it was logged.

Must do

• Any thread can call log(message) at any time
• Each entry appears in the file whole, never interleaved with another
• Entries appear in the order they were logged
• All accepted entries are written out on clean shutdown

Must hold

• Logging must not noticeably slow down the application threads calling it
• Throughput must keep up with today's peak load from all its threads