What it is
A load balancer spreads incoming requests across several servers, so you can add servers as traffic grows and lose one without an outage. Replication keeps copies of your data on several machines, for the same two reasons: more capacity for reads, and surviving a failure.
How it works
| Strategy | How it picks a server | Good for |
|---|---|---|
| Round robin | Next one in the list | Servers and requests that are all alike |
| Least connections | The one with the fewest open requests | Requests that take very different times |
| Hashing | The same key always goes to the same server | Keeping a user's data warm in one server's memory |
The balancer runs health checks and stops sending traffic to servers that fail them. It's easiest when app servers keep no state of their own, with sessions in a shared store, so any server can handle any request.
For data, the common setup is one primary that takes writes and copies them to replicas. If copying waits for replicas to confirm, it's synchronous: slower, but nothing's lost if the primary dies. If it doesn't wait, it's asynchronous: faster, but replicas run a little behind, and a crash can lose the last few writes.
Trade-offs
Reading from replicas spreads load, but a user might write something and then not see it, because their read hit a replica that hadn't caught up. When the primary fails, a replica gets promoted. Doing that automatically is fast, but if the old primary was only cut off, not dead, you can end up with two primaries.
In an interview
- Keep app servers stateless so you can add and remove them freely.
- Say whether replication is synchronous or asynchronous, and what each costs.
- Explain what happens when the primary dies.
Worked example
Surviving a database failure
A web app runs on 10 app servers behind a load balancer, with one primary database and two replicas. It handles 20,000 reads and 2,000 writes a second.
- The load balancer sends requests to whichever app server has the fewest in flight. Every few seconds it health checks each server, and a server that fails stops getting traffic.
- Sessions live in Redis, not on the app servers, so any server can handle any user. Adding an 11th server during a spike is just starting it.
- Writes go to the primary. Reads that can be slightly stale, like product pages, go to the replicas, which take most of the 20,000 reads a second.
- Replication is asynchronous, so replicas are usually a few milliseconds behind. Right after a user edits their profile, their reads go to the primary for a short while so they see their change.
- At 3 a.m. the primary's machine dies. Monitoring notices within seconds and promotes the replica that's furthest along to be the new primary.
- Any writes that hadn't reached that replica are lost. For most data that's a few milliseconds' worth. For payments, you'd make at least one replica synchronous so nothing confirmed is ever lost.
- The old primary is fenced off, so if it comes back it can't take writes and you don't end up with two primaries.
Follow-up questions
Things interviewers tend to ask once you've brought this up.
- What's the difference between layer 4 and layer 7 load balancing?
- Layer 4 looks at IP addresses and ports and forwards connections, which is fast and simple. Layer 7 understands HTTP, so it can route by path or header, end TLS, and retry failed requests.
- Can you have more than one primary?
- Yes, multi-primary or leaderless setups let writes go anywhere, which helps across regions. The cost is conflicts when two places change the same data, which you have to resolve.
- How do you route users across regions?
- DNS or anycast sends each user to the nearest region. If a region goes down, it's taken out of DNS and users go to the next closest one.
- What about sticky sessions?
- They send a user back to the same server every time. They're occasionally useful, like for WebSockets, but they make scaling and failures messier. Shared session storage is usually better.
Common mistakes
- Storing session state on app servers.
- Forgetting the load balancer itself needs to be redundant.
- Reading from replicas without thinking about stale reads.
- Not explaining what happens when the primary fails.
- Saying 'add replicas' to fix too many writes. Replicas help reads, and sharding helps writes.