Design an API rate limiter: A Sample System Design Solution

The system design question

Design a rate limiter for Stripe's API. Each user or key gets a limit, like 100 requests a minute, and it has to work across many servers. How does it scale, and what happens when the limiter's store is slow or down?

Agree on the scope (5 minutes)

Say up front what you're covering.

We'll design	We'll skip
Checking a key's limit on every API request	Network-level attack protection
Different limits per key and per endpoint	Monthly usage quotas and billing
Keeping one count across every API server	The admin UI for setting limits
What happens when the counter store is slow or down	Limiting how many requests run at once

Then agree on what "good" means:

Fast: the check adds a millisecond or two to a request, no more.
Shared: a key gets its limit across the whole fleet, not per server.
Close enough: a key might squeeze a few extra requests through now and then. That's fine.
Fails open: if the limiter breaks, the API keeps serving requests.
Clear to callers: a limited request gets a 429 and a header saying when to try again.

Rough numbers (5 minutes)

These are assumptions, but reasonable ones: 50,000 API requests a second at peak, about a million keys active in any given hour, and 100 API servers.

~50K
limit checks per second: ~1M
active keys: ~100
API servers: ~100 MB
for every key's counters

Question	Rough math	What it means
Can each server count on its own?	A 100 a minute limit spread over 100 servers	About 1 each, and load balancers don't spread one key's traffic evenly. We need a shared count
How much memory for the counters?	1M keys × about 100 bytes	About 100 MB. That fits in memory on one machine. Split it up for load and outages, not size
How much load on the shared store?	50K checks a second, one round trip each	A handful of in-memory store nodes handle that easily, at well under a millisecond each

The overall design (10 minutes)

One quick round trip to the counter store, before the request does any real work.

Every request goes through limiter middleware on the API server before it does anything else.
The middleware looks up the key's limits from config cached in memory. A key can have several, like 100 a minute overall and a tighter one for an expensive endpoint.
It makes one call to the counter store, which checks and updates the key's count in a single atomic step and says yes or no.
If yes, the request carries on, with headers saying how many requests are left. If no, it gets a 429 and a Retry-After header.
Every decision is counted in metrics, so we can see who's getting limited and spot when the limiter itself is struggling.

What the apps need to do

Action	Who does it	What's sent
Check a request	Limiter middleware, internal	Key, endpoint and cost (usually 1). Gets back allowed, how many are left and when it resets
Set a limit	Engineer or support	Key or key type, endpoint, rate and burst size
Rate limit headers	Sent back to the API client	Requests left in the window, and on a 429, how many seconds to wait

What we store and where

Data	Where it goes	Why
Counters	In-memory key-value store, split by API key	Sub-millisecond, atomic updates, and entries expire on their own
Limit config	Relational database, cached on every server	Small and rarely changed, so every server keeps a copy
Fallback counters	Each API server's own memory	Rough limits that keep working when the store doesn't

Step 1: Picking the algorithm (10 minutes)

There are four common rate limiting approaches. A fixed window counts per calendar minute. It's cheap, but a key can send 100 at 0:59 and 100 more at 1:00. A sliding log keeps every request's time, which is exact but stores 100 timestamps per key. A sliding window counter keeps this minute and last minute's counts and blends them, which is close and cheap.

I'd pick a token bucket. Each key has a bucket holding up to 100 tokens that refills at 100 a minute. A request takes a token, or is denied if there are none. It's two numbers per key: tokens left and when we last refilled. You don't need a timer: work out the refill each time from how long it's been.

It also matches how APIs get used. A client that's been quiet can send a short burst, but its average can't go over the limit. Different limits are just different buckets, and an expensive endpoint can take more than one token per request.

Step 2: Sharing the count across servers (10 minutes)

If two servers both read '1 token left', both allow, and both write 0, the key got one extra request. So the read, refill, take and write have to happen as one step inside the store. Most in-memory stores let you run a small script atomically, and that's what each check calls.

Use the store's clock for the refill math, not each server's, so servers with drifting clocks don't hand out extra tokens. Give each bucket an expiry of a few minutes, so idle keys clean themselves up.

Split keys across store nodes by a hash of the API key. That's sharding, and since one key always lives on one node, every check is a single round trip with nothing to coordinate. Keep the store in the same data center as the API servers and reuse connections, so a round trip stays well under a millisecond.

One huge customer

A key sending thousands of requests a second would hammer one store node. For keys like that, let each server grab a small batch of tokens, say 20, and hand them out from memory until they run out. It's a little less exact, but one key's traffic stops being one node's problem.

Step 3: When the store is slow or down (10 minutes)

Give the store call a tight timeout, a few milliseconds. If it times out or errors, the big choice is fail open or fail closed. For a payments API, fail open: a limiter outage shouldn't turn into an API outage. That's choosing availability over a perfect count, the same call as in CAP.

Failing open doesn't have to mean no limits at all. Each server also keeps a rough local bucket per key in memory, at the limit divided by the number of servers. When the store is unreachable, servers switch to those. One noisy client still can't run wild, and nobody gets a 500.

If lots of calls are timing out, stop calling the store for a few seconds and use local buckets, then try again. That stops a struggling store from getting buried by retries. Each store node has a replica, so a dead node fails over in seconds. The counts might reset when it does, which lets a few extra requests through. That's fine.

Alert when servers are running on local buckets, since limits are only rough while that's happening.
A few endpoints where abuse is costly, like ones that send emails or texts, can fail closed instead. Make that a setting per endpoint.

Choices you should be able to defend

We chose	Instead of	Because	We'd reconsider if
Token bucket	Sliding log	Two numbers per key, and it allows short bursts	Every request had to be counted exactly
One shared store, split by key	Each server counting alone	A key gets the same limit however traffic is spread	One round trip per request was too slow
Fail open, with rough local limits	Fail closed	A limiter outage doesn't take down the API	Letting extra requests through was dangerous or expensive
An atomic script in the store	Read, then write from the server	Two servers can't both take the last token	The store didn't support scripts
Token batches for huge keys	Always one round trip per request	One busy key can't overload a store node	We needed exact counts even for the biggest keys

What could go wrong (5 minutes)

What happens	What we do about it
The counter store is down	Servers switch to local buckets at a share of the limit, and an alert fires
The store is slow	The tight timeout trips, servers back off to local buckets for a few seconds, then retry
A store node fails over	Its keys move to the replica. Counts might reset, letting a few extra requests through
One key floods the API	It's denied with 429s at the API server, before its requests touch anything expensive
A bad limit is set, like 0 for everyone	Config changes are recorded and can be rolled back. Check the size of a change before saving it

Follow-up questions (5 minutes)

How would you limit requests running at the same time?

That's a concurrency limit. Add one to a counter when a request starts and take one off when it ends, with an expiry in case the server dies mid-request.

What about multiple regions?

Keep a store per region and give each region a share of the limit, or sync counts between regions every second or so. Exact global limits would need a cross-region call per request, which is too slow.

How do clients know they're close to the limit?

Send remaining-requests and reset-time headers on every response, and a Retry-After on a 429. Good client libraries back off on their own using those.

How the 60 minutes add up

Minutes	What you're doing
0–5	Agree on what you're building
5–10	Rough numbers
10–20	Draw the overall design
20–30	Picking the algorithm
30–40	Sharing the count across servers
40–50	When the store is slow or down
50–55	What could go wrong
55–60	Follow-up questions

System design: design an API rate limiter