System design: build a feature flag platform

Design a feature flag system for Stripe's engineering organization. Engineers can turn features on or off, roll them out to a percentage of merchants or users, and keep explicit include and exclude lists. It has to work across all of Stripe's backend services.

Say up front what you're covering.

Then agree on what "good" means:

• Nearly free to check: a flag check is an in-memory lookup, well under a millisecond, with no network call.
• Keeps working: if the flag service goes down, every service keeps using the last flags it saw.
• Quick to change: when someone flips a flag, every service sees it within a few seconds.
• Stable: a merchant in a 10% rollout gets the same answer on every request and in every service.
• Safe: every change is recorded, and a bad change can be undone in one click.

These are assumptions, but reasonable ones: 5,000 active flags, 500 services running on about 20,000 processes, a million flag checks a second across all of them, and a few hundred flag changes a day.

~5K

active flags

~20K

processes that check flags

~1M

flag checks per second

~10 MB

for every flag and rule

1. An engineer creates or edits a flag through the admin API. The flag store saves the new rules and bumps a global version number.
2. The change is published to a change feed, and a fresh full snapshot of all flags is written to file storage now and then.
3. A small agent on each host keeps an open connection to the flag service. When the version moves, it pulls the changed flags and writes them to local disk.
4. Each service uses a small library that loads the local copy into memory and answers is_enabled(flag, merchant) with no network call.
5. The library counts how often each flag is checked and reports it in the background, so we can spot flags nobody uses any more.

What the apps need to do

What we store and where

The check runs in order, and the first thing that matches wins: if the flag's switched off, return false. If the merchant's on the exclude list, false. If they're on the include list, true. Then the percentage rollout. If nothing matched, return the flag's default.

For the percentage, hash the flag name and the merchant ID together into a number from 0 to 9,999. If that number is below the rollout (500 for 5%), the flag's on. The same merchant always lands in the same bucket, so they get the same answer every time, in every service, without storing anything per merchant.

Two details matter here. Mixing in the flag name means different flags don't all pick the same 5% of merchants first. And when you go from 5% to 10%, everyone below 500 stays below 1,000, so nobody who had the feature loses it.

• Include and exclude lists load into a hash set, so checking one is a single lookup.
• Some lists get huge, like 'every merchant in Japan'. Make those named groups that live once and get shared by many flags, instead of copying 100,000 IDs into each one.
• If code asks for a flag that doesn't exist, maybe a typo, return the default written in the code and log a warning.

Each host runs one agent rather than every process talking to the flag service. That cuts 20,000 connections down to a few thousand. The agent sends the version it has and holds the request open until something changes, then gets just the changed flags. Changes land in a second or two.

The agent writes everything to local disk, and the library in each process reads from there. That's caching at every layer, and it's what keeps things working in an outage. If the flag service dies, agents keep the last copy and services keep checking flags as normal. They just don't see new changes until it's back.

That's a deliberate choice of availability over consistency, which is the CAP trade-off. For a few seconds after a change, one service might see the new value and another the old one. For flags that's fine. Make sure people know that, and don't use flags for anything that has to change everywhere at exactly the same moment.

A flag change can break production as easily as a code deploy can. So treat it like one: every change needs a reason, gets recorded with who made it, and creates a new version you can roll back to.

For risky flags, ramp up in steps, like 1%, 5%, 25% then 100%, and watch error rates between steps. Flags tagged as critical can require a second person to approve the change.

Old flags pile up. Use the check counts each library reports to find flags that have been at 100% for a month, and nag their owners to delete them.

• Caching strategiesMake reads faster, and figure out what to do when data changes.
• Consistency and the CAP theoremWhat happens when copies of your data can't agree.
• Sharding and consistent hashingSplit data across machines and keep it balanced.

We wrote this sample answer ourselves.

Practice Loop is independent and isn't affiliated with or endorsed by Stripe.