The system design question
Design a home page for a Confluence-like product that shows each user the most engaged-with pages across their organization, leaving out pages they wrote or have already read.
Agree on the scope (5 minutes)
Say up front what you're covering.
| We'll design | We'll skip |
|---|---|
| Counting views, likes and comments | The editor and page storage |
| Ranking pages per organization | Search |
| Removing pages you wrote or read | Personal interests and machine learning ranking |
| Only showing pages you're allowed to see | Notifications |
Then agree on what "good" means:
- Fast home page: it loads in a few hundred milliseconds.
- Fresh: a page that takes off this morning shows up within a few minutes.
- Personal: you never see pages you wrote or already read.
- Safe: you never see a page you don't have permission to open, not even its title.
Rough numbers (5 minutes)
All assumptions: 200,000 organizations, 50 million users with 10 million active a day, 500 million page views a day, and a few million likes and comments. Each person reads about 20 pages a day. The home page shows 20 pages.
- ~6K
- page views per second, on average
- ~30K
- page views per second at peak
- ~1,000
- home page loads per second
- ~1,800
- pages read per person in 90 days
| Question | Rough math | What it means |
|---|---|---|
| How much engagement comes in? | 500 million views a day ÷ 86,400 seconds | About 6K a second, maybe 30K at peaks, so count in a stream, not one database row per page |
| How much read history do we keep? | 50 million users × 1,800 page IDs × 8 bytes | About 700 GB, fine when split across machines |
| How many home page loads? | 10 million users × 8 visits a day ÷ 86,400 | Around 1,000 a second, each needing a per-person filter |
The overall design (10 minutes)
- Every view, like and comment becomes a small event on a stream, tagged with the organization, page, person and time.
- A counting job adds those up per page in hourly buckets and works out a score that favors recent activity.
- Every few minutes it saves each organization's top 500 pages, along with each page's author and space.
- Separately, each view is added to that person's read history.
- When you open the home page, we take your organization's top 500, drop pages you wrote, read or can't see, and show the first 20.
What the apps need to do
| Action | Who does it | What's sent |
|---|---|---|
| Get my home page | User | Nothing extra. Returns up to 20 pages with titles, authors and why they're trending |
| Record a view | Page viewer, sent by the app | Page ID. The server adds the user, organization and time |
| Like or comment | User | Page ID and the like or comment. This also sends an event |
| Hide this page | User | Page ID, so it never shows on their home page again |
What we store and where
| Data | Where it goes | Why |
|---|---|---|
| Hourly counts per page | The counting job's own store, split by organization | Enough to rebuild scores over the last 7 days |
| Top 500 pages per organization | Key-value store, keyed by organization | One read per home page load |
| Read history per person | Key-value store, split by user, last 90 days | To check 500 pages against what you've read |
| Pages, authors and permissions | The existing page database | To check who wrote a page and who can see it |
Step 1: Counting engagement (10 minutes)
Counting every view with a database update would mean 30K writes a second at peak, many on the same popular rows. Instead, send events to a stream and count them in a stream processing job that keeps running totals in memory and writes them out every minute.
Count in hourly buckets per page: views, unique viewers, likes and comments. A score could be something like unique viewers + 3 × likes + 5 × comments, with each hour's points shrinking the older they get. That way a page from this morning can beat one that was popular a week ago.
Count unique viewers, not raw views. Otherwise one person refreshing a page all day pushes it up. A small set of who's viewed it per page per day handles that.
To pick the top 500 per organization, keep a heap of the best 500 as you go through that org's pages. You don't need to sort every page.
Step 2: Removing what each person has seen (15 minutes)
This is the hard part. Everyone in an org shares one ranking, but each person has to filter it differently. Storing a separate ranked list per person would mean 50 million lists to keep updated. Don't. Keep one list per org, make it much longer than what we show, and filter when the page loads.
The list holds 500 pages, and we show 20. Filtering out your own pages is easy, because each entry already has its author. Filtering out pages you've read means checking 500 page IDs against your read history. Store that history as a set per user in a key-value store, and check all 500 in one batch call.
Then check permissions. A trending page in a private space must never show up for someone outside it, even as a title. Store each page's space with the list entry, and keep a cached list of spaces each person can see. Most pages pass or fail on space alone. Anything with special page-level restrictions gets a full permission check.
If filtering leaves fewer than 20, fall back to a longer time window or to recently updated pages in spaces you follow. Don't show an empty home page.
Making the read check cheaper
Keeping 90 days of reads is plenty, because a page you read four months ago is fine to see again. If read sets ever get too big, a bloom filter per person is a compact option. That's a small bit array that answers 'probably seen' or 'definitely not seen'. It can wrongly say you've read a page, which only means we skip one, but it never says you haven't read a page you have.
Caching the result
Cache each person's final 20 for a few minutes. The org list only changes every few minutes anyway. When you read a page, just drop it from your cached list rather than rebuilding everything.
Step 3: Very big and very small organizations (5 minutes)
A small team might have 30 pages and almost no activity. Its trending list would be noise. Use a longer window for small orgs, and fall back to 'recently updated' when there isn't enough activity to rank.
A huge org could send more events than one counting worker can handle. Split by organization and page rather than just organization, which is the same idea as sharding, then merge each split's top 500 into one.
Choices you should be able to defend
| We chose | Instead of | Because | We'd reconsider if |
|---|---|---|---|
| One ranking per org, filtered per person | A ranked list for every person | 50 million lists would be very costly to keep fresh | We added real personal ranking, like based on teams you work with |
| Top lists rebuilt every few minutes | Updating rankings on every event | Nobody notices a few minutes, and it's much cheaper | People needed to see breaking pages within seconds |
| Count unique viewers | Count every view | Stops refreshes and bots from gaming the list | Repeat views turned out to matter, like a page people keep going back to |
| 90 days of read history | Everything you've ever read | Bounded size, and old pages are fine to see again | People complained about seeing old pages |
What could go wrong (5 minutes)
| What happens | What we do about it |
|---|---|
| The counting job falls behind | Home pages keep showing the last saved list. It's a bit stale, but it still works |
| The read history store is slow | Skip the read filter with a short timeout and show the list anyway. Never skip the permission check |
| A page's permissions change after it was ranked | The permission check happens when the page loads, so it's always current |
| A bot views one page thousands of times | Unique viewers and per-user rate limits keep it from climbing |
| A page is deleted | Drop it during filtering. The next rebuild removes it from the list |
Follow-up questions (5 minutes)
How would you make it more personal?
Boost pages from spaces and people you interact with most. Keep the shared candidate list and change only the final ordering.
What if people want 'trending in my team' too?
Run the same counting per team or space, and keep a top list for each. It's the same machinery with a different key.
How would you know if it's any good?
Track how often people click home page items, and test score formulas against each other on a slice of users.
How the 60 minutes add up
| Minutes | What you're doing |
|---|---|
| 0–5 | Agree on what you're building |
| 5–10 | Rough numbers |
| 10–20 | Draw the overall design |
| 20–30 | Counting engagement |
| 30–45 | Removing what each person has seen |
| 45–50 | Very big and very small organizations |
| 50–55 | What could go wrong |
| 55–60 | Follow-up questions |
Key concepts used here
- Stream processing and windowingCount things in data that never stops coming in.
- Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.
- Caching strategiesMake reads faster, and figure out what to do when data changes.
- Heaps and priority queuesAlways have the smallest or largest item ready.
- Sharding and consistent hashingSplit data across machines and keep it balanced.
- Hash maps and sliding windowsSwap nested loops for a single pass.
Further reading
We wrote this sample answer ourselves.
Practice Loop is independent and isn't affiliated with or endorsed by Atlassian.