System design: deliver webhooks to merchants

Design the system that sends webhooks to merchants. When something happens in a merchant's account, like a payment succeeding, send an HTTP request to the URL they've registered. Deliver reliably, even when their servers are slow or down.

Say up front what you're covering.

Then agree on what "good" means:

• Never lose an event: every event gets delivered or clearly marked as failed.
• Quick: most webhooks should arrive within a few seconds.
• Isolated: one slow or broken merchant can't delay anyone else's webhooks.
• Secure: merchants can tell a webhook really came from us.

Made up, but reasonable: 500 million events a day, peaks at 5 times the average, about 10% of first attempts fail, and events kept for 30 days.

~6K

events per second, on average

~30K

events per second at a peak

10%

of first attempts fail

~30 TB

of events kept for 30 days

1. A service saves an event, like 'payment succeeded', in the same database update as the change itself.
2. A small job publishes saved events to a delivery queue, split up by merchant.
3. Delivery workers sign each event and POST it to every endpoint the merchant has set up for that event type.
4. Every attempt is recorded, along with the merchant's response.
5. Failed attempts go to a retry schedule, with longer and longer gaps between tries.

What the apps need to do

What we store and where

If a service saves a payment and then crashes before sending the webhook, the event's gone. So save the event in the same database update as the payment change. A separate job then publishes saved events to the queue. It might publish one twice, but it'll never lose one.

That means merchants can get the same event twice. Every event has a unique ID, and merchants should skip IDs they've seen. That's idempotency, and Stripe's docs tell merchants to do exactly that.

Workers sign each request with the merchant's secret, so they can check it's really from us. That's standard webhooks practice. Give each request a timeout of about 10 seconds.

The big risk is one slow merchant. If all workers are stuck waiting on them, everyone else waits too. So split delivery by merchant, a lot like sharding, and cap how many requests can be in flight to any one merchant. A slow merchant only slows themselves down.

Workers send requests without waiting on each one, so a single worker can have thousands in flight.

When an attempt fails, schedule the next one further and further out: a minute, then 5, 30, a few hours, and so on, for about three days. Add a little randomness so retries don't all land at once.

After the last try, mark the event failed, email the merchant, and let them resend it from the dashboard. If an endpoint fails constantly for days, turn it off and tell them.

• WebhooksHow one system tells another that something happened.
• Idempotency and retriesMake sure a retried request doesn't happen twice.
• Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
• Rate limitingStop too much traffic with token buckets and sliding windows.
• Sharding and consistent hashingSplit data across machines and keep it balanced.

We wrote this sample answer ourselves.

• Stripe docs: webhooks

Practice Loop is independent and isn't affiliated with or endorsed by Stripe.