Message queues and event streams

A message queue sits between the service creating work and the service doing it. One side drops off a message and moves on, and the other picks it up whenever it's ready. If traffic spikes or a consumer goes down, the messages just wait around instead of getting lost.

With a regular queue like RabbitMQ or SQS, a message goes away once someone's handled it. With a log like Kafka, it sticks around for a while. Each group of consumers keeps track of how far it's read, so a few different systems can read the same data, and you can go back and replay it.

Kafka splits each topic into partitions. Messages with the same key always land in the same partition, so they stay in order. You only get ordering inside a partition, though. More partitions also means more throughput.

Most of these systems can deliver a message twice, usually after a retry. The common fix is to give every message an ID and skip any ID you've already finished. Only confirm a message once the work is done, so if you crash halfway through, it gets retried and doesn't disappear.

• Say why you want one. Are you handling spikes, separating services, or sending the same thing to several places?
• Pick your partition key on purpose, and know what it does to ordering and busy partitions.
• Mention that messages can show up twice, and say how you'd deal with it.
• Don't forget it's one more thing to run, and it adds some delay.

Sending order confirmation emails

A store takes 200 orders a second at peak, and each one needs a confirmation email. The email provider takes about 300 ms per send and sometimes goes down for a few minutes.

1. If checkout sends the email itself, every order waits an extra 300 ms, and when the provider is down, checkout fails. That's a bad trade for an email.
2. So checkout writes an 'order placed' message to a queue and returns right away. Writing to the queue takes a few milliseconds.
3. Email workers read from the queue. At 300 ms per send, one worker handles about 3 a second, so you need around 70 workers to keep up with 200 a second. Call it 100 for headroom.
4. When the provider goes down for 5 minutes, about 60,000 messages pile up (200 × 300 seconds). Nobody's checkout breaks. When it comes back, the workers drain the backlog.
5. A worker might send an email and then crash before confirming the message, so it gets delivered again. Store the order ID of every email you've sent and skip ones you've seen.
6. A message that keeps failing, like a malformed email address, goes to a dead letter queue after a few tries so it doesn't block everything else.

Things interviewers tend to ask once you've brought this up.

What happens if a consumer is slower than the producer?

The queue grows. That's fine for a spike, but if it keeps growing you need more consumers, or more partitions so more consumers can work in parallel. Watch queue depth or consumer lag and alert on it.

How do you keep messages in order?

Only within a partition. Use a key like the order ID or user ID so everything for one entity lands in the same partition. Ordering across the whole topic means one partition, which means one consumer.

Can you get exactly-once delivery?

Not end to end in practice. You get at-least-once delivery plus an idempotent consumer, which acts like exactly once. Kafka has transactions for Kafka-to-Kafka pipelines, but side effects like sending an email still need deduplication.

Where does a message go when it keeps failing?

After a set number of retries, into a dead letter queue. Someone can look at it, fix the cause, and replay it. Without one, a single bad message can block a partition forever.

Why not just write to the database and have workers poll it?

That works at small scale and is easier to run. It gets painful with lots of workers fighting over rows and constant polling. A common middle ground is the outbox pattern: write the message to an outbox table in the same transaction, then publish it to the queue.

• Adding a queue without saying what problem it solves.
• Confirming a message before the work is done, so a crash loses it.
• Assuming each message arrives exactly once.
• Promising global ordering from a topic with many partitions.
• Picking a partition key that sends most traffic to one partition, like a country code when one country is 80% of traffic.

• Stream processing and windowingCount things in data that never stops coming in.
• WebhooksHow one system tells another that something happened.
• Sharding and consistent hashingSplit data across machines and keep it balanced.