Charge cards on a schedule: A Sample System Design Solution

The system design question

Design an internal system that captures money from a guest's card based on a per-booking setting, for example 50% at booking and 50% two days before check-in. It has to charge the right amount at the right time, even when things fail.

Agree on the scope (5 minutes)

Say up front what you're covering.

We'll design	We'll skip
Saving a payment plan for each booking	How the payment provider works inside
Running each charge at the right time	Refunds in detail
Charging exactly once, even with timeouts and crashes	Currency conversion
Handling failed charges and changed bookings	The guest's payment page

Then agree on what "good" means:

Exactly once: a guest is never charged twice for the same part, even if a worker crashes mid-charge.
The right amount: the parts always add up to the booking total, to the cent.
On time: a charge runs within a few minutes of when it's due, not hours late.
Nothing gets lost: every charge ends up either succeeded, failed for a known reason, or cancelled. Nothing sits there forever.

Rough numbers (5 minutes)

These are assumptions, made up but reasonable: 1.5 million bookings a day, 2 charges per booking on average, bookings made about 60 days before check-in, and the payment provider taking 1 to 3 seconds per charge and sometimes timing out.

~35
charges per second, on average: ~350
charges per second if they bunch up: ~90M
future charges waiting at any time: ~45 GB
of scheduled charges

Question	Rough math	What it means
How many charges a second?	1.5 million × 2 ÷ 86,400 seconds	About 35 on average. Bunched at midnight it could be 10 times that, so spread due times out
How many are waiting?	1.5 million × 60 days	About 90 million scheduled charges at any time
How much storage?	90 million × about 500 bytes	About 45 GB, which one relational database handles, split by booking later if needed
How many workers?	350 a second × up to 3 seconds each	About 1,000 calls in flight at the peak, so a few dozen workers each making many calls at once

The overall design (10 minutes)

Each booking's plan becomes one row per charge. A scheduler finds the ones that are due and queues them, and workers call the payment provider.

When a booking's made, the booking service sends the total and the plan setting, like '50% now, 50% two days before check-in'. We turn that into charge rows, each with an amount, a due time and a status of 'scheduled'.
The first charge, if it's due now, runs right away as part of the booking flow.
Every minute, a scheduler finds charges that are due and still scheduled, marks them 'queued' and puts them on a queue.
Charge workers take a charge, call the payment provider with a key that's unique to that charge, and record the result.
Failed charges get retried on a schedule, then go to the guest and hotel if they keep failing. Cancelled or changed bookings update their future charges.

What the apps need to do

Action	Who does it	What's sent
Create a payment plan	Booking service	Booking ID, total, currency, card token, plan setting, check-in date
Change or cancel a plan	Booking service	Booking ID, new total or dates, or cancel
Get a booking's charges	Support and finance tools	Booking ID. Returns each charge and its status
Provider result	Payment provider	Charge key, success or failure, provider reference

What we store and where

Data	Where it goes	Why
Charges (booking, amount, due time, status, attempts, key)	Relational database, split by booking ID	Status changes need transactions, and we query by due time
Index on (status, due time)	Same database	So 'what's due now?' is a quick range read
Charge attempts and provider replies	Same database, plus a long-term log	Finance needs a full history of every try
Due charges waiting for a worker	Message queue	Spreads work across workers and survives restarts

Step 1: Charging exactly once (12 minutes)

The scary case: a worker calls the provider, the call times out, and we don't know if the money moved. If we just try again, we might charge twice. If we give up, we might never charge.

The fix is a charge key. Every charge row gets a unique key when it's created, and we send it with every attempt. Most payment providers support a key like this, and a repeat with the same key returns the first result instead of charging again. That's idempotency, and it turns 'retry and hope' into 'retry safely'.

Track state carefully. A charge goes scheduled, then queued, then in progress, then succeeded or failed. Before calling the provider, the worker moves it to 'in progress' only if it's still 'queued', so two workers can't both pick it up. After the call, it saves the result in the same row.

If a worker crashes after calling the provider but before saving, the charge sits 'in progress'. A sweeper finds charges stuck there for more than a few minutes and retries them with the same key, or asks the provider for that key's status. Either way, no double charge.

Step 2: Running charges at the right time (10 minutes)

Don't create a timer per charge. With 90 million waiting, that's fragile and hard to see into. Store a due time on each row, and have a scheduler ask every minute for rows that are due and still scheduled, using the index on status and due time.

The scheduler marks each one 'queued' in the same step that picks it, then puts it on a queue. If two schedulers run at once, the status change means each charge only gets queued by one of them. If the queue loses a message, the sweeper notices a charge sitting in 'queued' too long and puts it back.

'Two days before check-in' needs a time zone. Use the hotel's local time, and spread charges across a window, like 9am to 5pm hotel time, rather than firing all of them at midnight. That flattens the peak and means a failure lands while support is awake.

Step 3: Amounts, failures and changes (8 minutes)

Work in whole cents. 50% of a €101.01 booking isn't a whole number of cents, so give the first parts the rounded-down share and make the last part whatever's left. That way the parts always add up to the total exactly.

Failures come in two kinds. Temporary ones, like a provider timeout or 'try again later', get retried with growing gaps. Final ones, like 'card declined', stop retrying and tell the guest to update their card, with a deadline before the booking's cancelled.

Bookings change. If the guest cancels, mark future charges cancelled in the same transaction. If the total changes, recalculate only the charges that haven't run. If the dates move, recompute due times. A charge that's already in progress finishes first, then the change applies.

Choices you should be able to defend

We chose	Instead of	Because	We'd reconsider if
Due times in a table, scanned every minute	A timer per charge	Easy to see, query and fix, and it survives restarts	We needed charges to the second, not the minute
A unique key per charge, sent to the provider	Checking our own records before retrying	Only the provider knows for sure if a timed-out call went through	The provider didn't support keys, so we'd have to query it before each retry
A relational database for charges	A key-value store	Status changes need transactions and we query by due time	Volume grew past what a split relational setup could handle
Spreading charges across the day	Charging at midnight	Flatter load, and failures happen while people are around to help	Hotels needed money at an exact time

What could go wrong (5 minutes)

What happens	What we do about it
The provider times out	Retry with the same charge key, so the provider returns the first result if it went through
A worker crashes mid-charge	The sweeper finds the charge stuck 'in progress' and retries it with the same key
The scheduler is down for an hour	Due charges pile up and run when it's back. Alert if anything's more than 15 minutes overdue
A card's declined	Stop retrying, ask the guest for a new card, and cancel the booking if the deadline passes
The guest cancels while a charge is in progress	Let it finish, then refund it if the cancellation policy says so

Follow-up questions (5 minutes)

How would you know it's working?

Track overdue charges, charges stuck in progress, failure rates by provider, and a daily check that each booking's charges add up to its total.

What if we switch payment providers?

Keep the provider behind one small interface, and save which provider handled each charge so retries and refunds go to the same one.

How would you support three or more parts?

The plan setting becomes a list of (percentage, when) pairs. The same rows, scheduler and workers handle any number.

How the 60 minutes add up

Minutes	What you're doing
0–5	Agree on what you're building
5–10	Rough numbers
10–20	Draw the overall design
20–32	Charging exactly once
32–42	Running charges at the right time
42–50	Amounts, failures and changes
50–55	What could go wrong
55–60	Follow-up questions

System design: charge cards on a schedule