The system design question
Design a system that collects ad click events and lets advertisers see click counts per ad over different time windows.
Agree on the scope (5 minutes)
Say up front what you're covering.
| We'll design | We'll skip |
|---|---|
| Taking in click events as they happen | Choosing which ad to show |
| Counting clicks per ad, per minute | Click fraud models in detail |
| Answering 'how many clicks in this time window' fast | Billing itself |
| Making the final numbers right for billing | Advertiser dashboards and charts |
Then agree on what "good" means:
- Fresh: counts are at most a minute or so behind.
- Right: each real click is counted once. The final numbers can be used for billing.
- Fast queries: any window, from the last hour to the last month, comes back in under a second.
- Doesn't lose clicks: a crash or a burst can delay counts, but never drop clicks.
Rough numbers (5 minutes)
These are assumptions, but reasonable ones: 10 billion clicks a day, peaks about 5 times the average, 1 million active ads, and each click event about 200 bytes.
- ~115K
- clicks per second, on average
- ~600K
- clicks per second at a peak
- ~2 TB
- of raw click events a day
- ~1.4B
- ad-minute counts a day, at most
| Question | Rough math | What it means |
|---|---|---|
| How many clicks a second? | 10 billion ÷ 86,400 | About 115K a second, and about 600K at a peak. Too many to write one by one to a normal database |
| How much raw data? | 10 billion × 200 bytes | About 2 TB a day, so raw clicks go to cheap storage |
| How many counts? | 1 million ads × 1,440 minutes | At most 1.4 billion rows a day. Most ads aren't clicked every minute, so it's far fewer |
| How big is a month of counts? | Say 200 million real rows a day × 50 bytes × 30 | About 300 GB. Fits fine in a store built for this kind of query |
The overall design (10 minutes)
- A click on an ad goes to our redirect service first. It gives the click an ID, writes the event to the click stream, and sends the user on to the advertiser's page.
- The click stream is split by ad ID, so all clicks for one ad go to the same place.
- An aggregator reads the stream, drops duplicate click IDs, and counts clicks per ad in 1-minute windows. When a window closes, it writes the counts to the counts store.
- Another reader copies every raw click to cheap storage. A nightly batch job recounts the day from there and overwrites the day's counts.
- The query service answers 'clicks for ad X between A and B' by adding up stored counts.
What the apps need to do
| Action | Who does it | What's sent |
|---|---|---|
| Record a click | User's browser, via the redirect link | Ad ID, impression ID, timestamp. Returns a redirect |
| Get click counts | Advertiser dashboard | Ad IDs, start time, end time, bucket size |
What we store and where
| Data | Where it goes | Why |
|---|---|---|
| Raw clicks | Event stream, then files in object storage | Huge, written once, read in bulk for recounts |
| Recent click IDs | In memory in each aggregator | Dropping duplicates only needs the last few minutes |
| Minute counts | Columnar or time-series store, keyed by ad ID and minute | Fast range sums over time for one ad |
| Hour and day counts | Same store, rolled up | Long windows add up a few hundred rows, not tens of thousands |
Step 1: Counting in a stream without losing or doubling clicks (12 minutes)
This is a stream processing job. The aggregator keeps a count per ad for the current minute, using the time the click happened, not the time it arrived. When the minute's over, it writes the counts out.
Clicks can arrive late, from slow phones or network trouble. So keep each window open a couple of minutes past its end before writing it. Clicks that are later than that still land in raw storage, and the nightly recount picks them up.
Duplicates come from browsers retrying and from the stream re-delivering after a crash. Each click has an ID, so the aggregator keeps the IDs it saw in the last few minutes and drops repeats. That's idempotency at the event level.
When the aggregator crashes
The aggregator saves a checkpoint every so often: where it is in the stream, plus its in-progress counts. After a crash, it restarts from the last checkpoint and replays from there.
Writes have to be safe to repeat. So write each minute's count as 'ad X, minute M = 812', not 'add 812 to ad X, minute M'. If a window gets written twice after a replay, the second write just sets the same number.
Step 2: One ad gets all the clicks (8 minutes)
Splitting the stream by ad ID keeps each ad's clicks together, which makes counting simple. But a viral ad or a Super Bowl spot can send more clicks to one partition than one aggregator can handle.
For ads we know are hot, add a small random number, 0 to 9 say, to the partition key. Their clicks spread over 10 partitions, each aggregator counts its slice, and a second small step adds the 10 partial counts for each minute. That's the usual sharding fix for a hot key.
Duplicate checks then need the same click ID to land on the same partition. So pick the random number from a hash of the click ID instead of truly at random. A retried click goes to the same place and gets caught.
Step 3: Answering any time window fast (10 minutes)
A month of minute counts for one ad is about 43K rows. Adding those up on every query is slow. So roll the counts up: a job adds minutes into hours and hours into days as each one closes.
A query for 'March 3, 10:15 to March 9, 16:40' then uses minute counts for the ragged ends, hour counts for the partial days, and day counts in the middle. That's a few hundred rows at most.
Store counts in a columnar or time-series store, sorted by ad ID and then time. A range of one ad's counts is a single sequential read. Cache answers for windows that are already closed, like yesterday, since they won't change until the nightly recount.
Fast numbers versus final numbers
The stream gives quick counts that are nearly right. The nightly batch job recounts from raw storage, with all the late clicks and with fraud filters applied, and overwrites the day's counts. Advertisers see fresh numbers during the day and exact ones for billing. Say up front that today's numbers can still shift a little.
Choices you should be able to defend
| We chose | Instead of | Because | We'd reconsider if |
|---|---|---|---|
| Counts per minute, rolled up | Storing every click and counting at query time | Queries read hundreds of rows, not billions | Advertisers needed to slice by things we don't pre-count, like device |
| Overwriting counts | Adding to counts | A replay after a crash writes the same value instead of doubling | Never |
| A nightly batch recount | Trusting the stream alone | Late clicks and fraud filters get applied before billing | The stream was proven exact and billing had to be real time |
| Salting only hot ads | Salting every ad | Most ads stay simple, with no merge step | Hot ads were hard to spot ahead of time |
What could go wrong (5 minutes)
| What happens | What we do about it |
|---|---|
| An aggregator crashes | Restart from the last checkpoint and replay. Overwrite writes make the replay harmless |
| The stream falls behind at a peak | Counts lag a few minutes. Add partitions and aggregators. The raw clicks are safe in the stream |
| The redirect service is slow | Users wait to reach the advertiser's site, which hurts. Write the event without waiting on anything else, and scale it out |
| A burst of bot clicks | Drop obvious bots at the redirect, and let the nightly recount apply full fraud filters |
| The counts store is down | The aggregator holds and retries its writes. Queries fall back to cached closed windows |
Follow-up questions (5 minutes)
How would you show the top 10 ads by clicks right now?
Have the aggregator keep a small heap of the top ads per partition each minute, then merge those into one top 10.
What if advertisers want counts by country or device too?
Add those as extra key fields in the minute counts. Each new field multiplies the rows, so only add the ones people really ask for.
How would you catch click fraud?
Look for lots of clicks from one IP or device, clicks with no matching ad impression, and clicks that come too fast after the ad was shown. Filter them in the nightly recount.
How the 60 minutes add up
| Minutes | What you're doing |
|---|---|
| 0–5 | Agree on what you're building |
| 5–10 | Rough numbers |
| 10–20 | Draw the overall design |
| 20–32 | Counting in a stream without losing or doubling clicks |
| 32–40 | One ad gets all the clicks |
| 40–50 | Answering any time window fast |
| 50–55 | What could go wrong |
| 55–60 | Follow-up questions |
Key concepts used here
- Stream processing and windowingCount things in data that never stops coming in.
- Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
- Sharding and consistent hashingSplit data across machines and keep it balanced.
- Idempotency and retriesMake sure a retried request doesn't happen twice.
- Heaps and priority queuesAlways have the smallest or largest item ready.
Further reading
We wrote this sample answer ourselves.
Practice Loop is independent and isn't affiliated with or endorsed by DoorDash.