The system design question
Design a donation app for a three-day fundraising campaign. Expect around 10 million users and $100 million in donations. Card payments go through a third-party provider.
Agree on the scope (5 minutes)
Say up front what you're covering.
| We'll design | We'll skip |
|---|---|
| The campaign page with a running total | Building our own card processing |
| Taking a donation and charging the card through the provider | Sign-up and accounts beyond an email |
| Making sure no donation is lost or charged twice | Refunds and chargebacks in detail |
| Receipts for donors | Tax rules |
Then agree on what "good" means:
- Never double charge: a donor who clicks twice, or retries after a timeout, pays once.
- Never lose a donation: if the provider took the money, we have a record of it.
- Stays up for the spike: the launch and any TV mention can bring huge bursts, and we only get three days.
- Honest total: the running total can lag by a few seconds, but it has to match the real money at the end.
Rough numbers (5 minutes)
These are assumptions, but reasonable ones: 10 million donations at about $10 each over three days, most traffic in a few big bursts at 50 times the average, and about 10 page views for every donation.
- ~40
- donations per second, on average
- ~2K
- donations per second at a peak
- ~20K
- page views per second at a peak
- ~10 GB
- of donation records in total
| Question | Rough math | What it means |
|---|---|---|
| How many donations a second? | 10 million ÷ about 260K seconds in three days | About 40 a second on average, so the average is easy. The peaks are the problem |
| How bad is a peak? | 40 × 50 | About 2K donations a second, which one well-sized database can still write |
| How much page traffic? | 2K × 10 views per donation | About 20K views a second, so the page needs to come from a cache, not the database |
| How much storage? | 10 million × about 1 KB | About 10 GB. Tiny. No need to split the data up |
The overall design (10 minutes)
- The campaign page is mostly static and served from a CDN. Only the running total changes, and the page fetches it from a cache every few seconds.
- When a donor opens the form, the browser makes a random donation key. Every retry of that donation sends the same key.
- The donation service saves a pending row with that key, then asks the provider to set up a payment for that amount, passing our donation ID as the provider's own duplicate-protection key.
- The donor types their card into the provider's payment form, so card numbers never touch our servers.
- The provider sends us a webhook when the payment succeeds or fails. We put it on a queue, mark the donation, add it to the total and send the receipt.
- A cleanup job checks donations stuck as pending and asks the provider what really happened.
What the apps need to do
| Action | Who does it | What's sent |
|---|---|---|
| Start a donation | Donor's browser | Donation key, amount, email, campaign ID |
| Get donation status | Donor's browser | Donation ID |
| Get the campaign total | Campaign page | Campaign ID |
| Payment result webhook | Payment provider, to us | Provider payment ID, our donation ID, status, signature |
What we store and where
| Data | Where it goes | Why |
|---|---|---|
| Donations | Relational database | Money needs transactions and a unique rule on the donation key. 10 GB fits on one machine |
| Webhook events | Relational database, plus a queue | Saved as they arrive, so a crash mid-processing doesn't lose one |
| Running total | In-memory cache, rebuilt from the database | Read 20K times a second, so it can't be a database sum |
| Campaign page | CDN | The same for everyone, so it can be cached at the edge |
Step 1: Never charging twice or losing a donation (15 minutes)
Think about what breaks. The donor double-clicks. Their phone times out and the app retries. Our server crashes right after the provider charged the card. Each of these either charges twice or leaves money with no record.
The fix is idempotency at both ends. The browser's donation key has a unique rule in the database, so a second request with the same key just gets back the first donation. Then, when we call the provider, we pass our donation ID as their duplicate-protection key. Most payment providers support that, and it means even if we call them twice, they only charge once.
Order matters too. Save the pending row first, then call the provider. If we crash after the charge, we still have the pending row, and we can ask the provider what happened. If we did it the other way round, a crash could leave a real charge we know nothing about.
Trusting the provider's answer
The donor's browser might say 'it worked', but we only mark a donation as paid when the provider tells us, through a signed webhook. Check the signature, save the event, then process it. Providers send the same webhook more than once, so processing has to skip events we've already handled.
Webhooks can also go missing. So every few minutes, a job looks for donations that have been pending for more than about 15 minutes and asks the provider for their status. At the end of the campaign, compare our records with the provider's payout report line by line.
Step 2: Surviving the spike (10 minutes)
Most of the traffic is people looking, not donating. Put the campaign page on a CDN so 20K views a second never reach our servers. The total is one small number, so cache it and let the page poll it every few seconds.
For donations, the donation service doesn't keep any state of its own, so we can run lots of copies. Since we know when the campaign starts, scale up before launch instead of waiting for autoscaling to catch up. Load test at a few times the expected peak the week before.
The provider has its own limits on how many calls we can make. Ask them for the limit ahead of time, and rate limit our calls to stay under it. If we hit it, put new donations in a short waiting line and show the donor a 'processing' screen rather than an error.
Webhook processing, receipts and the total can all fall behind for a minute without anyone being hurt. Put them behind a queue so a burst just makes the queue longer for a bit.
Step 3: The running total (5 minutes)
Summing 10 million rows on every page view won't work. Instead, keep a counter in the cache and add to it when a payment succeeds. One counter taking 2K updates a second is fine for an in-memory store. If it ever wasn't, split it into a few counters and add them up when reading.
The counter can drift if an update gets lost or applied twice. So every minute, a job sums successful donations in the database and resets the counter. The page might be a few seconds behind, which is fine. That's a consistency trade we're happy to make here.
Choices you should be able to defend
| We chose | Instead of | Because | We'd reconsider if |
|---|---|---|---|
| Provider's hosted card form | Collecting card numbers ourselves | We never touch card data, which saves a lot of security and compliance work | Never, for a three-day campaign |
| Pending row before charging | Charging first, then saving | A crash can't leave money we don't know about | Never |
| One relational database | A NoSQL store split across machines | 10 GB and 2K writes a second fit on one machine, and we want transactions and unique rules | The campaign was 100 times bigger |
| A cached total that lags a little | An exact total on every view | An exact sum can't handle 20K views a second | The total was legally binding in real time |
What could go wrong (5 minutes)
| What happens | What we do about it |
|---|---|
| The donor double-clicks or retries | Same donation key, so they get the first donation back, not a second charge |
| We crash right after the provider charges | The pending row is there. The webhook or the cleanup job marks it paid |
| A webhook never arrives | The cleanup job asks the provider about anything pending for too long |
| The provider is down | Show donors a friendly 'try again in a few minutes' message, and keep the page and total up |
| The database primary fails | Fail over to a replica that was kept in sync, so committed donations aren't lost |
Follow-up questions (5 minutes)
How would you handle recurring donations?
Save the donor's payment method with the provider, and run a scheduled job that starts a new donation each month with its own key, so a retried job doesn't charge twice.
What about refunds?
Add a refund record linked to the donation, call the provider's refund with its own duplicate-protection key, and subtract from the total when the provider confirms.
How would you stop card testing fraud?
Rate limit donations per IP and per card, add a challenge for suspicious traffic, and lean on the provider's fraud checks.
How the 60 minutes add up
| Minutes | What you're doing |
|---|---|
| 0–5 | Agree on what you're building |
| 5–10 | Rough numbers |
| 10–20 | Draw the overall design |
| 20–35 | Never charging twice or losing a donation |
| 35–45 | Surviving the spike |
| 45–50 | The running total |
| 50–55 | What could go wrong |
| 55–60 | Follow-up questions |
Key concepts used here
- Idempotency and retriesMake sure a retried request doesn't happen twice.
- WebhooksHow one system tells another that something happened.
- Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
- Rate limitingStop too much traffic with token buckets and sliding windows.
- Relational vs non-relational databasesPick a database based on how you'll actually use it.
- Consistency and the CAP theoremWhat happens when copies of your data can't agree.
- Load balancing and replicationSpread traffic across servers and keep copies so one failure isn't an outage.
Further reading
We wrote this sample answer ourselves.
Practice Loop is independent and isn't affiliated with or endorsed by DoorDash.