The system design question
Design a system where users pick their favorite color and connect with friends. Each user should see their friends' current favorite colors, and find out when a friend changes theirs.
Agree on the scope (5 minutes)
Say up front what you're covering.
| We'll design | We'll skip |
|---|---|
| Setting and changing your color | Sign-up and login |
| Adding and removing friends | Finding people to friend |
| Seeing all your friends' colors | Push notifications to phones |
| Finding out when a friend changes theirs | Blocking and privacy settings |
Then agree on what "good" means:
- Fast to open: your friends list with their colors loads in well under a second.
- Fresh enough: if you're online, you see a friend's change within a few seconds. If you're not, you see it next time you open the app.
- Never wrong for long: a friend's color can be a few seconds stale, but it should never stay wrong.
- Scales: it keeps working as users and friend lists grow, without a rewrite.
Rough numbers (5 minutes)
These are assumptions, so say them out loud: 100 million users, 10 million active each day, about 200 friends each (capped at 5,000), 5 million color changes a day, and each active user opens their friends list about 5 times a day.
- ~60
- color changes per second, on average
- ~12K
- friend updates per second to deliver
- ~120K
- colors read per second when lists open
- ~300 GB
- of friendships stored
| Question | Rough math | What it means |
|---|---|---|
| How many updates reach friends? | 5 million changes × 200 friends | About a billion a day, or 12K a second, maybe 60K at peaks |
| How many colors do we read? | 50 million list opens × 200 friends | About 120K color lookups a second, so reads dominate |
| How big is the friend graph? | 100 million users × 200 friends × about 16 bytes | Around 300 GB, too big for one machine to be comfortable |
The overall design (10 minutes)
- You pick a new color. The profile service saves it with the time it changed.
- The save publishes a small 'color changed' event to a stream.
- A notifier looks up which of your friends are online right now and pushes the change to them over their open connection.
- When someone opens the app, it loads their friends and their colors in one batch, and highlights any friend whose color changed since that person's last visit.
- Adding or removing a friend writes the friendship both ways, so either person can look it up quickly.
What the apps need to do
| Action | Who does it | What's sent |
|---|---|---|
| Set my color | User | The new color |
| Add or remove a friend | User | The friend's user ID |
| Get my friends and their colors | User | Optional page cursor. Returns each friend's color, when it changed, and whether it's new to you |
| Live updates | Us, to the app | An open connection that receives 'friend X changed to color Y' messages |
What we store and where
| Data | Where it goes | Why |
|---|---|---|
| Users and colors | Key-value store, split by user ID, with a cache in front | One small row per user, read constantly |
| Friendships | Wide table, split by user ID, one row per direction | So 'all of A's friends' is one quick read |
| Last visit time | On the user's row | To work out which friends changed since you last looked |
| Who's online and where | In-memory store with expiry | So the notifier knows which server holds each open connection |
Step 1: Getting changes to friends (15 minutes)
There are two basic ways to do this. Push means that when you change your color, we tell all your friends right away. Pull means each friend asks for the latest colors when they open the app. Push is quick to show but does work for people who might never look. Pull wastes nothing but feels slow if you're staring at the screen.
Here we can mix them cheaply. If a friend is online, push. If they're not, don't do anything at all. When they open the app, they load all their friends' colors anyway, and each color has a 'changed at' time. Compare it with their last visit and you know which ones are new. So there's no inbox to store for offline people.
For online people, the app holds an open connection, like a WebSocket, to one of many connection servers. A small in-memory store maps user ID to the server holding their connection. When an event comes off the stream, the notifier reads your friend list, checks which friends are online, and sends one message to each server that holds some of them.
Why a stream in the middle
Saving the color shouldn't wait on telling 200 people. The profile service writes the color and publishes an event, and it's done. If the notifier is slow or down, events pile up in the stream and go out when it recovers. Nothing's lost, and setting your color still feels instant.
Missing a live update
Live messages can get dropped if a connection blips. That's fine, because when the app reconnects it just reloads the friend list, which is always right. The live push is a nice extra, and the stored color is the truth.
Step 2: Storing friends and colors (10 minutes)
Friendships go in a table keyed by user ID, with one row per friend. Store both directions, so A's row lists B and B's row lists A. Writing two rows on every add is a small cost for making 'list my friends' a single read. Split this table across machines by user ID, which is sharding, so one user's friends all live together.
Colors are one tiny row per user. Opening the app means looking up 200 of them at once, and those friends are spread across many machines. So put a cache in front and fetch them in one batch. Most colors almost never change, so the cache hit rate will be very high.
When a color changes, update the database, then delete the cached copy. The next read fills it back in. A friend might see the old color for a moment, which we said was fine. That's the kind of consistency choice worth saying out loud.
A key-value or wide-column store fits better than a relational database here. We never join or run complex queries. It's all 'get this user's stuff'.
Step 3: People with lots of friends (5 minutes)
Someone with 5,000 friends who changes their color triggers 5,000 lookups. That's still fine for one event. The real risk is a lot of those at once. Split the friend list into chunks of a few hundred and hand each chunk to a different notifier worker, so one big list doesn't hog a worker.
If someone changes their color 50 times in a minute, there's no point telling friends 50 times. Rate limit changes per user, or only send the latest color once every few seconds.
Choices you should be able to defend
| We chose | Instead of | Because | We'd reconsider if |
|---|---|---|---|
| Push only to online friends | Push to everyone, with a stored inbox | Offline friends get the same answer by comparing times, with nothing extra to store | We needed a full history of changes, not only the latest color |
| Friendships stored both ways | One row per friendship | Listing your friends is one read on one machine | Storage got tight, which is unlikely at this size |
| Delete the cache on change | Update the cache directly | Simpler, and can't leave the cache with the wrong color for long | Reads right after a change got too slow |
| Key-value store | Relational database | Every query is by user ID, and it's easy to split | We needed queries like 'everyone whose color is blue' |
What could go wrong (5 minutes)
| What happens | What we do about it |
|---|---|
| The notifier is down | Events wait in the stream. Online friends see changes late, and anyone who reloads sees them right away |
| A connection server dies | Its users reconnect to another one and reload their friend list |
| The cache is empty after a restart | Reads fall through to the database for a bit. Warm it up slowly, or limit how many misses go through at once |
| Only one side of a friendship got written | Write both rows with a retry, and run a cleanup job that finds and fixes one-sided rows |
| A burst of changes, say on a holiday theme day | The stream absorbs it, and adding notifier workers catches up |
Follow-up questions (5 minutes)
How would you show a history of a friend's colors?
Keep a small log per user of their last few changes, with times. It's cheap because changes are rare.
What if friends could see only some people's colors?
Check visibility when building the friend list and when picking who to notify, using the same rule in both places.
What about phone notifications for offline users?
Add a separate consumer on the stream that batches changes per person and sends a daily summary, so you don't spam anyone.
How the 60 minutes add up
| Minutes | What you're doing |
|---|---|
| 0–5 | Agree on what you're building |
| 5–10 | Rough numbers |
| 10–20 | Draw the overall design |
| 20–35 | Getting changes to friends |
| 35–45 | Storing friends and colors |
| 45–50 | People with lots of friends |
| 50–55 | What could go wrong |
| 55–60 | Follow-up questions |
Key concepts used here
- Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
- Caching strategiesMake reads faster, and figure out what to do when data changes.
- Sharding and consistent hashingSplit data across machines and keep it balanced.
- Consistency and the CAP theoremWhat happens when copies of your data can't agree.
- Relational vs non-relational databasesPick a database based on how you'll actually use it.
- Rate limitingStop too much traffic with token buckets and sliding windows.
Further reading
We wrote this sample answer ourselves.
Practice Loop is independent and isn't affiliated with or endorsed by Atlassian.