Webhooks

A webhook is one system sending another system an HTTP request when something happens. Say you want to know when a payment goes through. You could keep asking, or you could give the payment provider a URL and let them tell you.

The sender POSTs the event as JSON, and retries if it doesn't get a success back. Your endpoint is on the public internet, so anyone could hit it. That's why providers usually sign each request using a secret only the two of you know. You work out the signature yourself and reject the request if it doesn't match.

• If you're sending them, talk about retries with backoff and where permanently failed events go.
• If you're receiving them, cover signatures, duplicates and responding quickly.
• Have a backup for missed events, like a job that checks for anything you didn't get.

Receiving payment webhooks

Your store uses a payment provider. When a payment succeeds, the provider sends you a webhook, and you mark the order as paid and start shipping it.

1. The provider POSTs a JSON event with an event ID, a type like payment.succeeded, and the payment details, plus a signature header.
2. Your endpoint recomputes the signature with your shared secret and compares. If it doesn't match, return 400 and stop. It also checks the timestamp is recent, so an old request can't be replayed.
3. It saves the raw event with its event ID under a unique constraint. If the insert fails because the ID exists, it's a duplicate: return 200 and do nothing else.
4. It puts the event on a queue and returns 200. All of this takes a few milliseconds, well inside the provider's timeout.
5. A worker picks it up, marks the order paid, and kicks off shipping. If shipping is down, the worker retries. The provider doesn't have to know.
6. Events can arrive out of order, like a refund before the payment. The worker checks the order's current state and the event's timestamp before applying it.
7. Once an hour, a job asks the provider's API for recent payments and compares them with your records, to catch any webhook that never arrived.

Things interviewers tend to ask once you've brought this up.

You're the sender. How do you handle a customer endpoint that's down?

Retry with exponential backoff and some randomness, over hours or days. After that, mark the delivery failed, alert the customer, and let them replay it from a dashboard. Keep one slow customer from holding up everyone else's deliveries.

How do you rotate the signing secret?

Support two secrets at once for a while. Sign with the new one, accept either, then retire the old one.

Should the webhook contain all the data, or just an ID?

Sending only an ID means the receiver calls your API to get the latest state, which avoids acting on stale or out-of-order data. Sending everything saves a call. Lots of providers send the full object and recommend fetching it anyway for important actions.

What if processing takes 30 seconds?

Don't make the sender wait. Return 200 as soon as it's safely stored, and process it in the background.

• Doing all the work before responding, so slow processing causes timeouts and retries.
• Skipping signature checks.
• Assuming each event arrives once and in order.
• Having no way to find events that never arrived.
• Retrying forever with no backoff when you're the one sending.

• Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
• Rate limitingStop too much traffic with token buckets and sliding windows.
• Consistency and the CAP theoremWhat happens when copies of your data can't agree.