Real-time delivery

Normally the client asks and the server answers. Chat messages, live prices and a driver moving on a map go the other way: the server has something new and needs to get it to the user right now, without waiting to be asked.

With WebSockets, each user holds a connection to one gateway server. A gateway can hold tens of thousands of mostly idle connections. When a user connects, you record which gateway they're on in a fast store like Redis. To deliver a message, look up the recipient's gateway and hand it the message, and the gateway writes it to their connection.

If they're not connected, save the message and send a mobile push notification. When they reconnect, the app asks for everything since the last message it saw.

Open connections are state. You can't just restart a gateway or move users around like you can with normal requests. Load balancers have to support long-lived connections, and a deploy disconnects everyone on that server, so clients need to reconnect on their own, with some random delay so they don't all come back at once.

• Say why polling isn't good enough before you add WebSockets.
• Explain how a message finds the server the recipient is connected to.
• Cover offline users and reconnecting, not just the happy path.

Delivering a chat message

A chat app has 10 million users online at peak. Alice sends Bob a message, and Bob should see it within a second if he's online, or get a notification if he isn't.

1. Each online user keeps a WebSocket open to a gateway server. At around 50,000 connections per gateway, 10 million users need about 200 gateways.
2. When Bob connected, his gateway wrote 'bob → gateway 37' into Redis, with an expiry it keeps refreshing while he's connected.
3. Alice's message goes over her WebSocket to the chat service, which saves it to the database first, with an ID that increases within the conversation.
4. The chat service looks up Bob in Redis, finds gateway 37, and sends it the message. Gateway 37 writes it down Bob's connection.
5. Bob's app sends back a 'delivered' acknowledgement, which updates the message's status and shows Alice the double tick.
6. If Bob isn't in Redis, he's offline. The chat service sends a mobile push notification through Apple or Google instead.
7. When Bob opens the app later, it reconnects and asks for every message after the last ID it has. That catches up anything a push missed.

Things interviewers tend to ask once you've brought this up.

How do you deliver to a group chat with 500 people?

Look up all 500 members' gateways, group them by gateway, and send each gateway one batch. For huge groups, put the message on a queue and let delivery workers fan it out.

What happens when a gateway restarts?

Its users disconnect and reconnect to other gateways, with a random delay so they don't all hit at once. They catch up using their last seen message ID, so nothing is lost.

How do you show who's online?

The same connection records give you presence. Don't push every status change to every friend, which is a flood. Send it to people who have the chat open, or let clients ask when they need it.

WebSockets or server-sent events?

If the server only pushes, like live prices or scores, server-sent events are simpler and work over plain HTTP. If the client sends a lot too, like chat or games, use WebSockets.

• Treating the push as the only copy of a message.
• Not explaining how a message finds the right gateway.
• Forgetting offline users and mobile push.
• Having every client reconnect at the same instant after a deploy.
• Using WebSockets for something that updates once a minute, where polling would be fine.

• Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
• Load balancing and replicationSpread traffic across servers and keep copies so one failure isn't an outage.
• Caching strategiesMake reads faster, and figure out what to do when data changes.