System design: tag content across products

Design a system that lets people add tags to content across Atlassian products, like Jira issues and Confluence pages. Users should see the tags on any item, and find everything with a given tag.

Say up front what you're covering.

Then agree on what "good" means:

• Fast both ways: showing an item's tags and listing a tag's items should both be quick.
• Safe: people should never see content they don't have access to.
• Quick to update: a new tag should show up within a second or two.
• Separate: each customer's tags stay in their own space.

Made up, but reasonable: 1 billion taggable items across all customers, about 3 tags each, and far more people reading tags than adding them.

3B

item-to-tag links

~300 GB

to store them, at about 100 bytes each

~2K

tag changes per second

~50K

tag look-ups per second

1. A product, like Jira, asks the tag service to add or remove a tag on one of its items.
2. The tag service saves the link and records the change.
3. Showing an item's tags or a tag's items reads straight from the tag database.
4. Each change goes on a queue, and a background job keeps tag counts and popular tags up to date.
5. Before showing results, we check the person can actually see each item.

What the apps need to do

What we store and where

Store every link twice: once organized by item, once organized by tag. That way both look-ups are a quick read. It costs some extra space and two writes per change, but tags change far less often than they're read.

Put the customer site in every key. Every query starts with the site, so one customer's data never mixes with another's, and later it gives us an easy way to split the data across machines.

A relational database is a fine start. If it gets too big, a wide-column store suits these 'give me everything under this key' look-ups well.

A tag like 'bug' can be on millions of items, so never return them all at once. Return a page at a time, and use a cursor that says where the next page starts.

If one tag gets too big for one place, split its items into a few buckets and read from each in turn.

Counting a tag's items on every request is too slow. A background job reads the change queue and keeps running counts in a cache, like stream processing. Popular tags are read a lot, so caching them helps too.

Each product owns its own permissions, and they change all the time. Copying them into the tag service is a good way to end up showing someone a page they shouldn't see.

So fetch a page of results, ask the product which ones the person can see, and drop the rest. If too many get dropped, fetch a bit more.

• Relational vs non-relational databasesPick a database based on how you'll actually use it.
• Sharding and consistent hashingSplit data across machines and keep it balanced.
• Caching strategiesMake reads faster, and figure out what to do when data changes.
• Stream processing and windowingCount things in data that never stops coming in.
• Message queues and event streamsKeep services from depending on each other, and handle traffic spikes.

We wrote this sample answer ourselves.

Practice Loop is independent and isn't affiliated with or endorsed by Atlassian.