Design a Rate Limiter

How to Use This Playbook

This playbook supports three reading modes:

What This Interview Actually Tests

Rate limiting is not an algorithm question. Everyone knows Token Bucket.

This is a distributed systems ownership question that tests:

• Whether you clarify intent before designing
• Whether you reason about failure modes proactively
• Whether you understand who pays for each tradeoff
• Whether you can own the operational burden

The key insight: Rate limiting is fundamentally a policy enforcement problem with no perfect answer. Staff engineers reason about who absorbs the cost of imperfection.

The L5 vs L6 Contrast (Memorize This)

Default Staff Positions (Unless Proven Otherwise)

The Three Intents (Pick One and Commit)

The Five Fault Lines (The Core of This Interview)

1. Protection vs Correctness — Do we prioritize protecting the system (allow drift) or enforcing exact limits (risk collapse)?

2. Centralized vs Distributed State — Redis per-request (simple, accurate, SPOF) vs local-first (resilient, fast, drifty)?

3. Latency vs Accuracy — Pay the 5-10ms Redis tax on every request, or accept approximation?

4. Fail-Open vs Fail-Closed — When the limiter fails, do we protect availability or protect resources?

5. Infra Ownership vs Team Autonomy — Central service vs sidecar vs SDK? Who owns policy changes?

Each fault line has a tradeoff matrix with explicit "who pays" analysis. See §3.

Quick Reference: What Interviewers Probe

Jump to Practice

→ Active Drills (§7) — 8 practice prompts with expected answer shapes

System Architecture Overview

Interview Walkthrough: How to Present This in 45 Minutes

This section bridges the gap between HelloInterview-style step-by-step guides and our Staff-level analysis. Senior candidates spend 25 minutes on the basics and run out of time before reaching anything interesting. Staff candidates speed through the baseline in 10-12 minutes — fast enough to spend the remaining 30+ minutes on the fault lines, failure modes, and ownership questions that actually determine your level.

The six phases below add up to 45 minutes. The ratios matter: phases 1-4 are deliberately compressed so phase 5 gets the lion's share of time. If you're spending more than 12 minutes before the transition to depth, you're pacing like an L5.

Phase 1: Requirements & Framing (2-3 minutes)

State functional requirements in 30 seconds — don't enumerate, state the category:

• "We need to limit request rates per client to protect backend services from abuse and enforce fair usage across tiers."

That's it. Don't list every edge case. The interviewer knows what rate limiting does.

Invest time on non-functional requirements (this is the Staff move):

• "What's the intent? Abuse protection, billing quota, or multi-tenant fairness? I'll assume abuse protection because that's where the hardest distributed tradeoffs live."
• Clarify: hard vs soft limits? Auth vs unauth traffic? Single vs multi-region?
• "For abuse protection, I want sub-5ms enforcement latency, fail-open behavior (I'll justify this), and eventual consistency across instances — I'll quantify the drift bound later."

Phase 2: Core Entities & API (1-2 minutes)

State entities quickly (30 seconds):

• RateLimitPolicy — tier, endpoint pattern, window size, threshold, action (reject / throttle / log)
• RateLimitCounter — composite key {api_key}:{endpoint}:{window}, count, TTL
• RateLimitDecision — allow/reject, remaining quota, retry_after_ms

Don't draw an ER diagram. Name the three nouns, confirm the interviewer is aligned, move on.

API (1 minute) — two surfaces:

Check path (hot, every request — middleware, not a standalone API):

CheckRateLimit(api_key, resource, action) → { allowed: bool, remaining: int, retry_after_ms: int }

Config path (cold, admin only):

PUT /rate-limits/rules   { tier, resource, limit, window, action }
GET /rate-limits/rules?tier=free
DELETE /rate-limits/rules/{rule_id}

Response headers on every request: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After

Phase 3: High-Level Architecture (5-7 minutes)

Draw three boxes and two data flows on the whiteboard:

┌──────────┐      ┌──────────────┐      ┌──────────────┐
│  Gateway  │─────▶│  Rate Limit  │─────▶│   Backend    │
│  / LB     │      │  Sidecar     │      │   Service    │
└──────────┘      └──────┬───────┘      └──────────────┘
                         │
                   local token bucket
                         │
                  periodic sync (async)
                         │
                  ┌──────▼───────┐
                  │    Redis     │
                  │ (coordination)│
                  └──────────────┘

Walk through the request flow: "Every request hits the sidecar, which checks a local token bucket — that's the hot path. The sidecar periodically syncs with Redis to coordinate across instances, but Redis is NOT in the critical path. If Redis is down, we fail-open with local counters as fallback."

Reference the full System Architecture diagram above for the complete multi-layer picture (CDN/WAF, config store, observability).

Key points to hit on the whiteboard:

1. Gateway-level enforcement — not per-service middleware (one enforcement point, not N)
2. Local-first with Redis coordination — local token bucket for sub-ms checks, periodic async sync to Redis for cross-instance coordination. Redis is NOT in the critical request path
3. Fail-open as default — for abuse protection, blocking legitimate users is worse than letting some abuse through (articulate why before the interviewer asks)
4. Config store separate from enforcement — policy changes propagate asynchronously, not in the hot path
5. Observability from day one — 429 rate, false positive rate, Redis latency p99, silent fail-open detection

Then immediately flag the key tension: "This gives us sub-millisecond checks at the cost of bounded drift across instances. For abuse protection, that's an acceptable tradeoff — I'll quantify the drift bound when we go deeper."

Phase 4: Transition to Depth (1 minute)

At this point you have a correct, simple architecture on the board. Now you pivot:

"The basic architecture is straightforward — gateway middleware + Redis counters. What makes this Staff-level is the failure mode reasoning. Let me dive into three areas: (1) what happens when Redis fails, (2) distributed coordination across multiple gateway instances, (3) policy management as an organizational problem."

Then offer the interviewer a choice:

"I can go deep on any of these. Which is most interesting to you?"

If the interviewer doesn't have a preference, lead with fail-open vs fail-closed — it's the most impressive and the most universally applicable.

Phase 5: Deep Dives (25-30 minutes)

The interviewer will steer, but be prepared to go deep on any of these. For each, follow the Staff pattern: state the tradeoff → pick a position → quantify the cost → explain who absorbs that cost.

Fault Line 1: Fail-open vs fail-closed (5-7 min)

Open with the tradeoff framing:

"When Redis is down, do we let all traffic through (fail-open) or block all traffic (fail-closed)? For abuse protection, I default to fail-open: blocking 100% of legitimate users to stop potential abuse is worse than temporarily allowing unchecked traffic. For billing/quota, I'd flip to fail-closed because giving away resources has direct revenue impact."

Go deeper — walk through the failure sequence:

1. Redis goes down → middleware detects failure (connection timeout or error response)
2. Middleware switches to local in-memory counters (degraded accuracy but non-zero enforcement)
3. Observability pipeline fires alert: "enforcement rate dropped below 95%"
4. On-call engineer sees alert, confirms Redis outage, decides whether to intervene
5. Redis recovers → middleware detects healthy connection → resumes centralized counters

The real danger: silent fail-open. If the fallback silently passes all traffic without alerting, you could run unprotected for hours. Cross-reference §3 Fault Lines and §4 Failure Modes for the full analysis.

Fault Line 2: Distributed coordination — local vs centralized counters (5-7 min)

Frame the problem with concrete numbers:

"With 10 gateway instances and a 100 req/min limit, each instance could independently allow 100 — giving the client 1000 total. The options are:

• (a) Centralized Redis on every request — accurate, +2ms latency per check
• (b) Local counters with periodic sync — fast (sub-ms), bounded drift
• (c) Pre-split quotas: 100/10 = 10 per instance — no coordination needed but wastes capacity on cold instances"

Pick a position and quantify: "I'd go with option (b) for abuse protection. With 10 instances syncing every 5 seconds, worst case overshoot is 10 × (100/60 × 5) ≈ 83 extra requests per window. That's an 83% overshoot in the absolute worst case — but in practice, traffic distributes across instances, so real overshoot is 10-20%. For abuse protection where limits are 1000+ req/min, that's noise."

Then show you know when to switch: "For billing/quota where every request has dollar value, I'd switch to option (a) — centralized Redis. The +2ms latency is acceptable because billing endpoints are lower throughput, and accuracy matters more than latency."

Fault Line 3: Algorithm selection — why it matters less than you think (3-5 min)

"I'd use token bucket. But honestly, the algorithm choice is the least interesting part of this problem. Token bucket, sliding window log, sliding window counter — they all work. The real question is: where does the counter live, what happens when that store fails, and how do you coordinate across instances. I can explain the algorithmic differences if you'd like, but I'd rather spend time on the distributed coordination problem."

This is a power move. It demonstrates you know the algorithms but won't waste time on textbook recitation. If the interviewer insists, give a 30-second summary:

• Token bucket: smooth, allows bursts up to bucket size, O(1) per check
• Sliding window counter: approximation between fixed windows, low memory, slight inaccuracy at boundaries
• Sliding window log: exact, but O(n) memory per client — doesn't scale for high-volume clients

Then redirect: "The algorithm determines local behavior. The hard problem is distributed coordination — which we just discussed."

Hot keys & thundering herd (3-5 min)

"What happens when a single API key generates 50% of all traffic? That key's counter becomes a hot key in Redis — every gateway instance contends on the same key. The mitigations: (a) local aggregation — batch increments locally and flush to Redis every 100ms instead of per-request, (b) key sharding — split rl:{api_key}:{window} into rl:{api_key}:{window}:{shard_0..7} and sum on read, (c) early rejection — if a key is already 10x over limit, reject locally without hitting Redis at all."

This topic shows you've operated rate limiters at scale — hot keys are a production problem, not a design problem.

Ownership & policy management (3-5 min)

"Who writes the rate limit policies? In my experience, this is where rate limiting actually breaks. The platform team owns the enforcement infrastructure, but product teams own the policies for their endpoints. Without a self-service policy API and a review process, you end up with either: (a) the platform team as a bottleneck for every policy change, or (b) product teams setting limits too high (because they fear blocking users) and the limits being effectively useless."

The Staff answer: "Self-service policy API with guardrails. Product teams can set limits within pre-approved ranges. Changes go through a review pipeline — not a human review, but an automated check that the new limit won't exceed the backend's capacity. Deployment is canary: new limits apply to 5% of traffic for 1 hour before full rollout."

Operational maturity (3-5 min)

"How do you detect silent fail-open? If Redis goes down at 3 AM and the rate limiter silently stops enforcing, how long until someone notices?"

Name three concrete signals:

1. Enforcement rate metric: % of requests checked against Redis vs local fallback — alert when < 95%
2. 429 rate anomaly: if 429s drop to zero during a traffic spike, something is wrong
3. Redis health check: connection pool errors, latency p99 > 10ms, replication lag

"The on-call runbook has three steps: (1) check Redis cluster health, (2) if Redis is down, verify local fallback is active, (3) if local fallback is also failing, escalate to incident — we're running unprotected."

Phase 6: Wrap-Up (2-3 minutes)

Summarize the key tradeoff — don't just restate your architecture, synthesize the insight:

"Rate limiting is a policy enforcement problem, not an algorithm problem. The Staff-level challenge is: who absorbs the cost of imperfection? For abuse protection, we bias toward fail-open because blocking legitimate users is worse than letting some abuse through. For billing, we bias toward fail-closed because giving away resources has direct cost. The architecture is the same in both cases — the configuration and failure mode behavior change."

If time permits, add the organizational insight:

"The harder problem is policy management. The rate limiter is infrastructure — it's a solved technical problem. The unsolved problem is getting 15 product teams to agree on rate limit policies, keep them updated, and actually respond when limits are hit. That's an organizational design problem, not a systems design problem."

Common Timing Mistakes

Reading the Interviewer

What to Deliberately Skip

These topics are traps. L5 candidates spend time on them. Staff candidates name them, dismiss them, and redirect to what matters.

The pattern: acknowledge you know it, state your position in one sentence, redirect to the interesting problem. This is how you buy time for the depth that actually differentiates you.

11. The Staff Lens

1.1 Why This Problem Exists in Staff Interviews

This is NOT an algorithm question. Everyone knows Token Bucket.

This is a Distributed Systems & Operational Ownership question that tests:

• Whether you clarify intent before designing
• Whether you reason about failure modes proactively
• Whether you understand who pays for each tradeoff
• Whether you can own the operational burden

1.2 The L5 vs L6 Contrast

Recall the five key behaviors from the Executive Summary. Below, we explain why each matters and what interviewers listen for.

Behavior 1: First move (clarify intent before architecture)

Behavior 2: Algorithm choice (avoid the latency trap)

Behavior 3: Consistency (be explicit about partial correctness)

Behavior 4: Failure handling (replicas don't answer degraded mode)

Behavior 5: Ownership (avoid client-library hell)

1.3 The Key Insight

22. Problem Framing & Intent

2.1 The Three Intents

Before drawing any boxes, ask Why? The implementation changes entirely based on intent:

This sentence alone separates L5 from L6.

2.2 What's Intentionally Underspecified

The interviewer deliberately avoids specifying:

• Auth vs unauth traffic
• Client identity strength
• Hard vs soft limits
• Multi-region behavior
• Regulatory constraints

Staff engineers surface these unknowns. Senior engineers assume them away.

2.3 How to Open (The First 2 Minutes)

1. Ask 1-2 clarifying questions about intent
2. State your assumption explicitly
3. Outline your plan: placement → semantics → coordination → failure modes → observability

Example opening:

2.4 Terminology (Use Precise Words)

Rate-limiter interviews are ambiguous about where enforcement runs. Use precise terms:

33. The Five Fault Lines

This section contains the Staff-grade tradeoff reasoning. Each fault line includes:

• A tradeoff matrix
• Explicit "who pays" analysis
• L6 vs L7 calibration
• Bar-raiser follow-up questions

3.1 Fault Line 1: Protection vs Correctness

The tradeoff: Strict correctness requires coordination that can become the bottleneck. Protection-first accepts drift but keeps the system alive.

L6 (Staff) answer: Picks the priority explicitly based on intent. For abuse protection, chooses protection (availability) with bounded drift, aggressive timeouts, and clear mitigations (local fallback caps + alerting + circuit breaker when backend shows stress).

L7 (Principal) answer: Reframes as risk governance: "Which layer enforces what?" (CDN/WAF for coarse abuse, gateway for identity-aware limits, app for business invariants). Defines who signs off on fail-open/closed and what blast radius is acceptable.

3.2 Fault Line 2: Centralized vs Distributed State

The tradeoff: Central state is easy to reason about but creates a dependency. Distributed state is resilient but requires explicit coordination mechanisms.

L6 (Staff) answer: Chooses one concrete coordination mechanism and explains it (token leasing, key-owner routing, or bounded reconciliation). Names the exact failure mode they're preventing (multi-writer race, hot key QPS, partial enforcement) and the explicit tradeoff.

L7 (Principal) answer: "Do we even need custom distributed coordination?" Evaluates managed gateway throttling, Envoy RLS, CDN/WAF. If custom is needed, selects mechanism based on operational cost and governance.

→ For coordination mechanism details, see →Appendix C: Storage & Coordination Patterns

3.3 Fault Line 3: Latency vs Accuracy

The 10ms Trap:

• Central Redis adds 5-10ms to every request
• At 1M req/sec, this is massive infrastructure cost
• Local-first reduces latency but introduces drift

L6 (Staff) answer: Quantifies the "latency tax" and chooses an architecture that avoids putting a flaky dependency in the critical path. Defines what "acceptable drift" means for abuse protection and where billing needs stricter semantics.

L7 (Principal) answer: Connects latency to business outcomes: "Which requests deserve the tax?" Separates paths: strict centralized checks only for billing-critical endpoints; fast-path for abuse. Adds a cost model.

→ If you choose "hybrid" coordination, review →Appendix C for mechanism details (leasing, routing, reconciliation).

Key:   {identity}:{scope}     // e.g., "user:123:api/orders"
Value: {tokens, last_refill}  // e.g., {47, 1699999999}
TTL:   refill_window          // e.g., 60s for per-minute limit

3.4 Fault Line 4: Fail-Open vs Fail-Closed

Decision Framework:

L6 (Staff) answer: Makes the decision quickly, then immediately lists mitigations and observability (timeouts, conservative local fallback caps, bypass-rate alerting, circuit breaker if backend shows stress).

L7 (Principal) answer: Defines a governance model: who can flip fail-open/closed, what is the emergency procedure, what is the kill-switch scope, and what post-incident analysis is required.

Guardrails for Fail-Open:

• Aggressive Redis timeout (5-10ms max)
• Conservative local fallback caps
• Bypass-rate alerting (if bypass_rate > threshold, page on-call)
• Circuit breaker on backend stress signals

→ For the complete decision framework, see →Degraded Mode Framework — applies to circuit breakers, feature flags, and dependency isolation.

3.5 Fault Line 5: Infra Ownership vs Team Autonomy

L6 (Staff) answer: Recommends enforcing at the Gateway/Sidecar to avoid per-service SDK drift. Calls out rollout and ownership boundaries: limits should be deployable and observable centrally, while product teams can express policy intent.

L7 (Principal) answer: Defines governance: who owns policy definition, who approves changes, how you do staged rollouts, and how you prevent the platform team from becoming a bottleneck.

Staff Choice: Gateway/Sidecar — decouples infrastructure from business logic, avoids "client library hell."

44. Failure Modes & Degradation

→ This section applies the →Degraded Mode Framework. Review it if you need the full fail-open/fail-closed decision tree.

4.1 Store Failures

Scenario A: Central Store Becomes Slow (Most Common)

Timeline:

t=0:     Redis p99 jumps from 2ms to 100ms
t=0-30s: Gateway worker threads pile up waiting
t=30s:   Gateway request queues fill
t=1min:  Gateway starts returning 503s
t=2min:  "Rate limiter" has become a global outage

What breaks first: P99 latency at the gateway spikes, threads pile up, rate limiter becomes latency amplifier.

Bad reaction: "Increase Redis timeouts" — makes it worse.

Staff reaction:

• Redis timeout: 5-10ms max
• On timeout: bypass limiter (fail-open)
• This is a deliberate fail-open decision with alerting

Scenario B: Central Store Is Down

Staff choice for abuse protection:

• Fail-open with aggressive local limits
• Alert on bypass rate
• Circuit breaker on backend stress

4.2 Hot Key & Amplification

Why "just shard it" doesn't work:

Hashing distributes different keys across shards, but a single identity is still one key. If that identity dominates traffic, it will dominate one shard.

Scenario A: Leaked API Key Used by Botnet

• Symptom: One API key drives 50%+ of traffic; single shard CPU spikes
• Mitigation:
  1. Immediate deny-cache locally (TTL 30-60s)
  2. Revoke/rotate the key
  3. Add CDN/WAF edge rules if stable IP/ASN signals
• Tradeoff: Fast containment vs false positives if key is shared by legitimate partner

Scenario B: Legitimate Tenant Burst (Partner Batch Job)

• Symptom: Paying tenant looks like "hot key" but isn't malicious
• Mitigation:
  1. Token leasing with adaptive lease sizing for higher-tier tenants
  2. Per-tenant reservation + shared burst pool
• Tradeoff: Fairness vs utilization

Scenario C: IP-Based Identity Collapses (NAT/Corporate Proxy)

• Symptom: One IP = thousands of real users → false positives
• Mitigation:
  1. IP as coarse outer limiter only
  2. Shift to stronger identity (API key / user id) when possible
• Tradeoff: Better UX vs implementation complexity

Mitigations that actually work:

1. Reduce remote operations per request — Token leasing or bounded batch reconciliation
2. Key-owner routing — Single-writer for hot identity
3. Deny-cache / blocklist escalation — Short-circuit locally, then escalate
4. Separate abuse from quota — Different correctness bars, different paths

→ For mechanism details on leasing and routing, see →Appendix C or the standalone →Distributed Coordination Framework.

4.3 Data Integrity Failures

Clock Skew

Token refill depends on time. Drift causes over-refill or under-refill.

Mitigations:

• Cap refill deltas (never trust large time jumps)
• Use monotonic clocks
• Consider Redis server time inside Lua scripts

Script Bugs

Atomic operations silently wrong. Hardest to detect.

Detection: Integration tests, audit logging Recovery: Script rollback, counter reset

4.4 Operational Reality Matrix

55. Evaluation Rubric

5.1 Level-Based Signals

5.2 Strong Hire Signals

5.3 Lean No Hire Signals

5.4 Common False Positives

• Knows Redis deeply: Deep Redis knowledge ≠ good system design
• Draws complex diagrams: Complexity isn't a Staff signal
• Mentions many algorithms: Breadth without depth is Senior, not Staff

66. Interview Flow & Pivots

6.1 Typical 45-Minute Structure

6.2 How Interviewers Pivot

6.3 What Silence Means

• After tradeoff question: Interviewer wants you to reason aloud
• After "what else?": You're missing something important
• After definitive answer: They may disagree or want nuance

6.4 Follow-Up Questions to Expect

1. "How do you handle clock skew?"
2. "What if a single user generates 90% of traffic?"
3. "How do you test this in production?"
4. "What metrics would you monitor?"
5. "How do you handle a global rate limit across regions?"
6. "What's your failure budget for this service?"

77. Active Drills

Practice these scenarios to internalize Staff-level thinking. Try answering before revealing the Staff approach.

→ For the complete framework, see →Build vs Buy Framework.

Prompt: "Product wants a new limit tomorrow. How do you ship it safely?"

These scenarios test Staff-level operational thinking. Unlike drills (which test interview responses), deep dives test ownership reasoning — the kind of thinking that happens when you're the Staff engineer responsible for the system.

Deep Dive 1: Flash Sale Incident

Deep Dive 2: Silent Fail-Open

Deep Dive 3: Large Tenant Onboarding

Deep Dive 4: Post-Mortem — Limiter Let Through an Attack

Deep Dive 5: Multi-Region Expansion

99. Level Expectations Summary

What gets you each level in a rate limiting interview:

What Separates Each Level

Quick Self-Check

Before your interview, verify you can answer:

• What are the three intents, and how does each change the design?
• What is the latency trap, and how do you avoid it?
• When would you fail-open vs fail-closed, and who signs off?
• What breaks when one identity is 90% of traffic?
• How do you ship a policy change without breaking top tenants?

The Bar for This Question

Mid-level (L4/E4): You should define clear API endpoints (POST /limit-check) and land on a working design with token bucket or sliding window. You can explain why rate limiting exists, implement basic per-user limiting with a Redis counter, and handle the 429 response correctly. Deep dives into distributed counting or multi-tier policies would be a bonus but aren't expected.

Senior (L5/E5): You should quickly build the baseline architecture and spend meaningful time on distributed rate limiting — Redis-based counting with MULTI/EVAL, the consistency-availability tradeoff of the counter (is approximate counting acceptable?), and fail-open vs fail-closed behavior. You should have an opinion on sliding window vs token bucket for your use case and be able to explain the latency implications of a synchronous Redis call on every request. Landing on a local-counter-with-periodic-sync approach for latency-sensitive paths would be strong.

Staff+ (L6/E6+): You should breeze through the architecture in under 5 minutes and spend 25+ minutes on depth: multi-tier rate limiting (per-user, per-tenant, per-service, global), the organizational negotiation of who sets rate limit policies and who gets exceptions, failure mode analysis (what happens when Redis is unavailable — do you fail-open and risk abuse, or fail-closed and risk an outage?), and how rate limiting intersects with capacity planning and cost attribution. You should reason about the "top tenant" problem — one customer consuming 90% of quota — and propose policy governance. The interviewer should learn something from your answer.

1010. Staff Insiders: Controversial Opinions

These are uncomfortable truths that distinguish Staff engineers from Seniors. They're based on operating rate limiters at scale, not on textbook knowledge. Strong engineers disagree on some of these — that's the point.

1010.1 "Exact Rate Limiting" Is a Myth

The uncomfortable truth: At scale, you are never enforcing the limit you think you're enforcing.

Why it's a lie:

The Staff position: Stop pretending you're enforcing "exactly 1000 req/s." You're enforcing "approximately 1000 req/s ± ε." The Staff question is: what's your ε, and is it acceptable for your intent?

Why this matters in interviews: Candidates who claim "exact enforcement" without acknowledging drift reveal they haven't operated rate limiters at scale. The bar-raiser question is: "What's the worst-case over-admission in your design, and who signed off on it?"

1010.2 Abuse Protection and Billing Cannot Share an Algorithm

The uncomfortable truth: If you're using the same rate limiter for abuse protection and billing enforcement, one of them is wrong.

Why they conflict:

The Staff position: These are fundamentally different systems. Trying to serve both with "one rate limiter" leads to:

• Billing drift (abuse limiter is too loose)
• Availability problems (billing limiter is too strict for abuse)
• Operational confusion (one team's change breaks the other)

Real-world signal: Companies that conflate these eventually have an outage where the "rate limiter" either (a) let through an attack because it was tuned for billing, or (b) rejected paying customers because it was tuned for abuse. Then they split them.

1010.3 Global Fairness Dies at Scale (And That's OK)

The uncomfortable truth: Many companies that claim "global rate limiting" are lying. At true global scale, they've abandoned it.

Why global coordination fails:

The dirty secret: At hyperscale, many companies enforce "per-region limits" and call it "global." A user with a 1000 req/min global limit might actually get 1000 × N (where N = number of regions) if they distribute traffic.

The Staff position: Global fairness is a spectrum, not a binary. The honest question is: "What's the blast radius of our approximation, and is that acceptable?"

When to abandon global fairness:

• Cross-region latency exceeds your p99 budget
• Coordination failures cascade to availability problems
• The cost of global accuracy exceeds the cost of over-admission

The bar-raiser question: "If a sophisticated user figures out your per-region limits and distributes traffic across 5 regions, what happens? Is that acceptable?"

t=0.99s: 100 requests → Allowed
t=1.01s: 100 requests → Allowed
Result: 200 requests in ~20ms

Refill rate: 100 tokens/sec
Bucket capacity: 200 tokens (burst)
Cost per request: 1 token (default)

t=0.0s: 150 requests → 150 allowed (tokens: 200→50)
t=0.5s: 120 requests → Refilled 50 tokens (50→100)
                     → 100 allowed, 20 rejected (429)
t=0.6s: 10 requests  → Refilled ~10 tokens (0→10)
                     → 10 allowed (tokens: 0)

rate_limit:{identity}:{endpoint}:{window}

rate_limit:ip:203.0.113.42:/login:60s
rate_limit:apikey:abc123:/search:10s
rate_limit:user:u_789:/api/v1:1m

Gateway A: READ tokens=50
Gateway B: READ tokens=50
Gateway A: WRITE tokens=49
Gateway B: WRITE tokens=49
Result: Two requests consumed one token

-- KEYS[1] = bucket key
-- ARGV[1] = capacity
-- ARGV[2] = refill_rate_tokens_per_sec
-- ARGV[3] = now_ms
-- ARGV[4] = cost_tokens
-- Returns: {allowed, remaining, retry_after_ms}

HTTP/1.1 429 Too Many Requests
Retry-After: 2
Content-Type: application/json

{"error":"rate_limit_exceeded","retry_after_seconds":2}

HTTP/1.1 200 OK
X-RateLimit-Remaining-Tokens: 42
X-RateLimit-Refill-Rate: 100
X-RateLimit-Burst-Capacity: 200

HTTP/1.1 429 Too Many Requests
Retry-After: 2
X-RateLimit-Reset-After: 2
X-RateLimit-Remaining-Tokens: 0

rate_limit_allowed_total
rate_limit_rejected_total
rate_limit_bypassed_total
rate_limit_latency_ms
redis_latency_ms
redis_timeout_total

rate_limit_allowed_total{path,identity,endpoint}    # Request allowed
rate_limit_rejected_total{path,identity,endpoint}   # Request rejected (429)
rate_limit_bypassed_total{path,identity,endpoint}   # Limiter failed, bypassed

refill_rate[tenant] = R_global * (weight[tenant] / W_total)

These frameworks are referenced throughout this playbook and apply to many system design problems:

• →Distributed State Coordination

  • Token leasing, key-owner routing, bounded reconciliation
  • Applies to: rate limiting, caching, locks, leader election, sessions

• →Degraded Mode Framework

  • Fail-open vs fail-closed decision tree
  • Applies to: rate limiting, circuit breakers, feature flags, dependency isolation

• →Build vs Buy Framework

  • TCO analysis, managed vs custom decision
  • Applies to: rate limiting, observability, auth, CDN, API gateway, mesh