The system design question
Design the payment system for an online game. Players buy in-game credits with a credit card, spend credits on items, and can get refunds. Everything has to stay correct under heavy concurrency and partial failures.
Agree on the scope (5 minutes)
Say up front what you're covering.
| We'll design | We'll skip |
|---|---|
| Buying credits with a card | The card processing itself (we use a payment provider) |
| Each player's credit balance and history | The item catalog and pricing |
| Spending credits on items | Fraud models |
| Refunds, back to credits or back to the card | Taxes |
Then agree on what "good" means:
- Never wrong: nobody gets charged twice, credits never go negative, and every item that's paid for gets delivered.
- Fast spending: buying an item with credits should feel instant, well under half a second.
- Survives failures: a crash or a timeout halfway through leaves things in a state we can finish or undo.
- Auditable: you can explain every balance from its history, and it matches what the payment provider says.
Rough numbers (5 minutes)
These are assumptions, made up but reasonable: 50 million players a day, 2 million card purchases a day, 200 million credit spends a day, and peaks about 10 times the average when a big sale or event drops.
- ~25
- card purchases a second, on average
- ~2.3K
- credit spends a second, on average
- ~25K
- credit spends a second at a peak
- ~45 TB
- of history a year
| Question | Rough math | What it means |
|---|---|---|
| How many card purchases? | 2 million ÷ 86,400 seconds, then × 10 at a peak | About 250 a second at most. Low volume, but each one waits on an outside provider |
| How many spends? | 200 million ÷ 86,400, then × 10 | About 25K a second at a peak, all inside our own systems |
| How much history? | 200 million spends × 2 entries × about 300 bytes × 365 days | About 45 TB a year, so the history has to be split across machines |
The overall design (10 minutes)
- The client sends every request with a unique request ID it made up, so retries and double taps can be spotted.
- To buy credits, the payments service records a 'purchase started' row, then asks the payment provider to charge the card.
- When the provider confirms, through its reply or a webhook, the service adds the credits to the player's balance. It can only do that once per purchase.
- To spend credits, the service subtracts the price and writes the history entry and an 'item bought' event, all in one database transaction.
- The inventory service reads 'item bought' events from the queue and gives the player the item. A refund runs the same steps in reverse.
What the apps need to do
| Action | Who does it | What's sent |
|---|---|---|
| Buy credits | Player's client | Credit pack, request ID |
| Spend credits | Player's client or game server | Item ID, expected price, request ID |
| Request a refund | Player or support agent | Purchase or spend ID, reason, request ID |
| Get balance and history | Player's client | Player ID, page of results |
| Payment update webhook | Payment provider, to us | Charge ID, new status, signature |
What we store and where
| Data | Where it goes | Why |
|---|---|---|
| Wallets | Relational database, split by player | One row per player with their balance and a version number |
| Ledger entries | Same database, same split | Append-only history of every change, so balances can always be rebuilt and checked |
| Card purchases | Same database | Where each purchase is in its life: started, charged, credited, refunded or failed |
| Request IDs | Same database, unique per player | So a repeated request returns the first result and doesn't run again |
| Outgoing events | Same database, then the queue | Written in the same transaction as the money change, then published |
| Inventory | Inventory service's own database | Which items each player owns, keyed by the spend that paid for them |
Step 1: Buying credits with a card (12 minutes)
The scary part is the call to the payment provider. It can time out, and then you don't know if the card was charged. So never call it before you've written down that you're about to. First save a purchase row marked 'started', with its own purchase ID. Then call the provider, and pass that purchase ID as the provider's idempotency key. If we retry, the provider sees the same key and won't charge again.
Treat the purchase as a little state machine: started, then charged or failed, then credited. Each step only moves forward from the step before it. Adding credits checks that the purchase is 'charged' and flips it to 'credited' in the same transaction as the balance change, so it can't happen twice even if the provider's reply and its webhook both show up.
Most providers also send webhooks when a charge settles or fails later. Check the signature, look up the purchase by its ID, and run the same 'move to charged' step. Whoever arrives second finds nothing left to do.
When you don't know what happened
If the provider call times out, leave the purchase as 'started'. A background job picks up purchases stuck there for more than a few minutes and asks the provider what happened to that ID. Then it finishes the purchase or marks it failed. The player sees 'processing' in the meantime, which is much better than being charged with nothing to show for it.
Once a day, compare our purchases with the provider's settlement report. Anything that's in one and not the other goes to a person to look at. This catches the bugs nobody thought of.
Step 2: Spending credits under heavy concurrency (10 minutes)
Picture a player tapping 'buy' on two items at the same moment with only enough credits for one. If both requests read the balance, both see enough money and both go through. The fix is to make the check and the subtraction one step: 'take 100 credits from this wallet only if the balance is at least 100'. The database locks the row while it does that, so the second request sees the new balance and fails cleanly.
In that same transaction, write two ledger entries (credits leave the player, credits arrive in the game's revenue account), save the request ID, and write an 'item bought' event. Either everything is saved or nothing is. That's the same database guarantee you'd use for any money movement, and it's why we pick a relational database here. See relational vs non-relational.
Split the data by player ID, which is sharding. A spend only touches one player's wallet, so it's a single-machine transaction, and different players never block each other. The game's revenue account would be a hot spot if every spend updated one row, so give it many sub-accounts, or just add up its ledger entries later.
Delivering the item
The inventory service reads 'item bought' events from a queue. It might see the same event twice, so it stores the spend ID with the item and ignores repeats. If it can't give the item at all, say the item was retired, it publishes a failure and the payments service refunds the credits automatically.
Step 3: Refunds (8 minutes)
There are two kinds. An item refund puts credits back and takes the item away. It's just a spend in reverse: one transaction that adds credits, writes the ledger entries and sends a 'take item back' event. If the item's already been used up, the policy decides, usually no refund.
A card refund is harder because it involves the outside provider again. You can only refund credits the player hasn't spent. So first take those credits out of their balance and park them in a 'refund pending' state. Then ask the provider to refund the card, again with a request ID. If it succeeds, the purchase becomes 'refunded'. If it fails for good, put the credits back.
Doing it in that order matters. If you refunded the card first and then tried to take the credits, the player could spend them in between, and you'd have given away items for free.
Choices you should be able to defend
| We chose | Instead of | Because | We'd reconsider if |
|---|---|---|---|
| One relational database, split by player | A NoSQL store | Transactions across balance, history and events in one step make correctness simple | A single player's activity was too much for one machine, which is unlikely |
| Strong consistency for balances | Eventually consistent balances | A stale balance lets players overspend | Never, for money. Stale reads are fine for the history screen |
| Events written in the same transaction as the money change | Calling the inventory service directly | A crash can't leave a player charged without an event to deliver their item | We needed the item to show up in the same response, though a short wait is usually fine |
| Take credits first, then refund the card | Refund the card first | The player can't spend credits that are about to be refunded | Never |
What could go wrong (5 minutes)
| What happens | What we do about it |
|---|---|
| The provider times out mid-charge | The purchase stays 'started' and a background job asks the provider what happened |
| The client retries a spend after a network blip | Its request ID is already saved, so it gets the first result back and nothing runs twice |
| The inventory service is down | Events wait in the queue. The player's credits are already spent, and the item shows up when it recovers |
| A database machine fails | A replica takes over. We use synchronous replicas for wallets, so no committed spend is lost |
| A player gets a chargeback from their bank | Take the credits back. If they've spent them, let the balance go negative and block spending until it's paid |
Follow-up questions (5 minutes)
How would you handle promotional or bonus credits?
Track credits in lots, paid and free, and spend free ones first. Then a card refund only ever covers paid credits the player still has.
What about a player hammering the buy button with a bot?
Request IDs stop duplicates, and rate limiting per player stops floods before they reach the database.
How do you support more than one currency?
Price credit packs per currency and store the amount and currency on each purchase. Credits themselves stay one currency inside the game, so the spend side doesn't change.
How the 60 minutes add up
| Minutes | What you're doing |
|---|---|
| 0–5 | Agree on what you're building |
| 5–10 | Rough numbers |
| 10–20 | Draw the overall design |
| 20–32 | Buying credits with a card |
| 32–42 | Spending credits under heavy concurrency |
| 42–50 | Refunds |
| 50–55 | What could go wrong |
| 55–60 | Follow-up questions |
Key concepts used here
- Idempotency and retriesMake sure a retried request doesn't happen twice.
- Consistency and the CAP theoremWhat happens when copies of your data can't agree.
- Relational vs non-relational databasesPick a database based on how you'll actually use it.
- Sharding and consistent hashingSplit data across machines and keep it balanced.
- Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
- WebhooksHow one system tells another that something happened.
- Rate limitingStop too much traffic with token buckets and sliding windows.
Further reading
We wrote this sample answer ourselves.
Practice Loop is independent and isn't affiliated with or endorsed by Databricks.