Design a Distributed Rate Limiter

How to Use This Playbook

Organized for interview use first, reference second. Read front-to-back once. Return to individual sections for targeted review.

Without rate limiting:
t=0:      "50% off everything" push notification
t=+10s:   Traffic spikes 10x — 50,000 req/s hits the API
t=+30s:   Database connection pool exhausted
t=+45s:   Backend returning 500 errors
t=+2min:  Full outage. All users see error pages.
t=+45min: Engineers restore service. Revenue lost. Trust damaged.

With rate limiting:
t=0:      Same push notification, same 10x spike
t=+10s:   Rate limiter kicks in — excess traffic gets 429 responses
t=+10s:   Backend stays at healthy capacity (5,000 req/s)
t=+1min:  Traffic normalizes as retries spread out
t=+5min:  Zero downtime. Graceful degradation. A metric blip, not a page.

What This Interview Actually Tests

Rate limiting is not an algorithm question. Everyone knows Token Bucket.

This is a distributed systems ownership question that tests:

• Whether you clarify intent before designing
• Whether you reason about failure modes proactively
• Whether you understand who pays for each tradeoff
• Whether you can own the operational burden

The key insight: Rate limiting is fundamentally a policy enforcement problem with no perfect answer. Staff engineers reason about who absorbs the cost of imperfection.

The L5 vs L6 Contrast — Start Here

The Staff Positions

The Three Intents

Three intents drive every design decision. Each leads to a fundamentally different architecture.

The Five Fault Lines

In the Wild: Real Production Systems

Stripe — Local-First Token Bucket with Centralized Billing

Stripe uses a local-first token bucket for abuse protection at the gateway layer, keeping enforcement sub-millisecond. For billing/quota (tracking API call usage against paid plans), they use a separate centralized path with stronger consistency guarantees. Every response includes RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset headers — treating rate limit transparency as a first-class API contract.

Figma — Backpressure-Based Throttling

Figma's collaborative editing infrastructure uses backpressure-based throttling rather than hard 429 rejection — graceful degradation through reduced cursor update fidelity, delayed sync batching, and selective feature throttling under load. When a document has too many simultaneous editors, the system reduces real-time sync frequency rather than disconnecting users.

Cloudflare — Edge-First Multi-Layer Defense

Cloudflare enforces rate limiting at multiple layers: L3/L4 filtering at the network edge for volumetric DDoS, WAF-level rules at L7 for application-aware limits, and challenge pages as an intermediate step before hard blocks. Each layer catches what the previous one misses, with different accuracy/latency tradeoffs at each tier.

What Interviewers Probe

System Architecture Overview

Quick-Reference: The 30-Second Cheat Sheet

Key Numbers Worth Memorizing

Phase 1: Requirements & Framing (2–3 minutes)

State functional requirements in 30 seconds — don't enumerate, state the category:

"We need to limit request rates per client to protect backend services from abuse and enforce fair usage across tiers."

Invest remaining time on non-functional requirements — this is the Staff move:

"What's the intent? Abuse protection, billing quota, or multi-tenant fairness? I'll assume abuse protection because that's where the hardest distributed tradeoffs live."

Then commit to a constraint set: "For abuse protection: sub-5ms enforcement latency overhead, fail-open behavior (I'll justify this), and eventual consistency across instances — I'll quantify the drift bound. These three constraints drive the entire design."

Phase 2: Core Entities & API (1–2 minutes)

State entities quickly — 30 seconds:

• RateLimitPolicy: tier, endpoint pattern, window size, threshold, action (reject / throttle / log)
• RateLimitCounter: composite key {identity}:{endpoint}:{window}, count, TTL
• RateLimitDecision: allow/reject, remaining quota, retry_after_ms

Don't draw an ER diagram. Name the three nouns, confirm alignment, move on.

Check path (hot, every request — middleware, not a standalone API):

CheckRateLimit(identity, resource, action) → { allowed: bool, remaining: int, retry_after_ms: int }

Config path (cold, admin only):

PUT /rate-limits/rules   { tier, resource, limit, window, action }
GET /rate-limits/rules?tier=free
DELETE /rate-limits/rules/{rule_id}

Response headers on every request: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After

Phase 3: High-Level Architecture (5–7 minutes)

Draw the architecture in three layers, then walk through the request flow:

Walk the request flow in 90 seconds:

1. Every request hits the gateway's rate limit middleware
2. Middleware checks local token bucket — this is the hot path, ~0ms
3. If tokens available: allow, forward to backend
4. If no tokens: return 429 + Retry-After header
5. Background: every 500ms, sync local count deltas to Redis for cross-instance coordination
6. Redis is never in the critical request path

Key points to hit explicitly:

1. Gateway-level enforcement — one enforcement point, not N per-service middlewares
2. Local-first with Redis coordination — local token bucket for sub-ms checks; async sync to Redis; Redis is NOT in the critical path
3. Fail-open as default — for abuse protection, blocking legitimate users is worse than letting some abuse through
4. Config store separate from enforcement — policy changes propagate asynchronously, not in the hot path
5. Observability from day one — 429 rate, false positive rate, Redis latency p99, bypass rate

Phase 4: Transition to Depth (1 minute)

"The basic architecture is straightforward. What makes this Staff-level is the failure mode reasoning. Three areas worth going deep: what happens when Redis fails, distributed coordination across multiple gateway instances, and policy management as an organizational problem. Which is most interesting to you?"

If the interviewer doesn't have a preference: lead with fail-open vs fail-closed — most impressive and most universally applicable.

Phase 5: Deep Dives (25–30 minutes)

For each, follow the Staff pattern: state the tradeoff → pick a position → quantify the cost → name who absorbs it.

Fault Line 1: Fail-open vs fail-closed (5–7 min)

Open with the decision framework:

"When Redis is down, do we let all traffic through (fail-open) or block all traffic (fail-closed)? For abuse protection, I default to fail-open: blocking 100% of legitimate users to stop potential abuse is worse than temporarily allowing unchecked traffic. For billing/quota, I'd flip to fail-closed because giving away resources has direct revenue impact."

Walk through the failure sequence:

1. Redis goes down → middleware detects failure (connection timeout, 5–10ms aggressive timeout)
2. Middleware switches to local in-memory counters — degraded accuracy but non-zero enforcement
3. Observability pipeline fires alert: "enforcement rate dropped below 95%"
4. On-call engineer sees alert, confirms Redis outage, follows runbook
5. Redis recovers → middleware detects healthy connection → resumes centralized counters

The real danger — silent fail-open: "If the fallback silently passes all traffic without alerting, you could run unprotected for hours. This is the scenario most Senior candidates miss. Adding a bypass-rate metric (rate_limit_bypassed_total) with an alert threshold is mandatory, not optional."

Fault Line 2: Distributed coordination (5–7 min)

Frame with concrete numbers:

"With 10 gateway instances and a 100 req/min limit, each instance could independently allow 100 — giving the client 1,000 total. Three options:

• Centralized Redis per-request: accurate, +2ms latency per check, SPOF
• Local counters with periodic sync: fast (sub-ms), bounded drift
• Pre-split quotas (100/10 = 10 per instance): no coordination, wastes capacity on cold instances"

Pick a position and quantify: "I'd go with option (b) for abuse protection. With 10 instances syncing every 5 seconds, worst-case overshoot is 10 × (100/60 × 5) ≈ 83 extra requests per window. That's ~83% overshoot absolute worst case — but in practice traffic distributes across instances, so real overshoot is 10–20%. For abuse protection where limits are 1,000+ req/min, that's noise."

Name a concrete coordination mechanism — don't just say "sync with Redis":

• Token leasing: Lease N tokens from Redis, spend locally. Reduces Redis QPS by 10–100×. Failure mode: stranded tokens when gateway crashes mid-lease.
• Key-owner routing: Hash the identity to a consistent gateway owner. Single-writer avoids races. Failure mode: load skew if one identity dominates.
• Bounded reconciliation: Explicit drift budget ε; force global check when local tokens drop below low-watermark. Failure mode: thundering herd on low-watermark trigger.

"For billing/quota where every request has dollar value, I'd switch to centralized Redis. The +2ms latency is acceptable because billing endpoints are lower throughput."

Fault Line 3: Algorithm — why it matters less than you think (3–5 min)

"I'd use token bucket. But the algorithm choice is the least interesting part. Token bucket, sliding window log, sliding window counter — they all work. The real question is where the counter lives, what happens when that store fails, and how you coordinate across instances. I can explain the algorithmic differences if you'd like, but I'd rather spend time on the distributed coordination problem."

This is a power move. It demonstrates you know the algorithms but won't waste time on textbook recitation. If the interviewer insists:

• Token bucket: smooth, allows bursts up to bucket size, O(1) per check
• Sliding window counter: approximation between two fixed windows, low memory, ±1% error at boundaries
• Sliding window log: exact, but O(N) memory per client — doesn't scale for high-volume clients

Then redirect: "The algorithm determines local behavior. The hard problem is distributed coordination — which we just discussed."

Fault Line 4: Hot keys & thundering herd (3–5 min)

"What happens when a single API key generates 50% of all traffic? That key's counter becomes a hot key in Redis — every gateway instance contends on the same key. The mitigations:

• (a) Local aggregation: batch increments locally and flush to Redis every 100ms instead of per-request
• (b) Key sharding: split rl:{api_key}:{window} into rl:{api_key}:{window}:{shard_0..7} and sum on read
• (c) Deny-cache: if a key is already 10× over limit, reject locally without hitting Redis at all"

The important distinction: "Sharding Redis helps for many different keys across nodes. It does NOT help for a single hot identity that hashes to one shard. Hot key is structurally different from high cardinality — you need local aggregation or key-owner routing, not just more shards."

Fault Line 5: Ownership & policy management (3–5 min)

"Who writes the rate limit policies? In my experience, this is where rate limiting actually breaks. The platform team owns the enforcement infrastructure, but product teams own the policies for their endpoints. Without a self-service policy API and a review process, you end up with either: the platform team as a bottleneck for every policy change, or product teams setting limits too high (because they fear blocking users) and the limits being effectively useless."

The Staff answer: "Self-service policy API with guardrails. Product teams can set limits within pre-approved ranges. Changes go through a review pipeline — not a human review, but an automated check that the new limit won't exceed the backend's capacity. Deployment is canary: new limits apply to 5% of traffic for 1 hour before full rollout. Rollback is a one-line config change, not a deploy."

Phase 6: Wrap-Up (2–3 minutes)

Synthesize the insight — don't restate the architecture:

"Rate limiting is a policy enforcement problem, not an algorithm problem. The Staff-level challenge is: who absorbs the cost of imperfection? For abuse protection, we bias toward fail-open because blocking legitimate users is worse than letting some abuse through. For billing, we bias toward fail-closed because giving away resources has direct cost. The architecture is the same in both cases — the configuration and failure behavior change."

The organizational closer:

"The harder problem is policy management. The rate limiter is infrastructure — it's a solved technical problem. The unsolved problem is getting 15 product teams to agree on rate limit policies, keep them updated, and actually respond when limits are hit. That's an organizational design problem, not a systems design problem."

Common Timing Mistakes

1The Staff Lens

1.1 Why This Problem Exists in Staff Interviews

Rate limiting separates L6 from L5 because it forces you to reason about organizational tradeoffs — who absorbs the cost of imperfection, who owns the policy, who gets paged at 3 AM. Five behaviors below are what interviewers listen for.

1.2 The L5 vs L6 Contrast — Visual

1.3 The Staff Question That Cuts Through Everything

This single question reveals whether a candidate has operated rate limiters in production or only designed them theoretically.

2Problem Framing & Intent

2.1 The Three Intents — Explained

Abuse Protection → fail-open, speed-first

• Constraint: latency overhead must be sub-5ms; enforcement must not become a kill switch
• Algorithm: token bucket, local-first with async coordination
• Failure mode: fail-open with conservative local caps and bypass-rate alerting
• Who pays for imperfection: security/product team (explaining why some requests got through during Redis outage)

Billing/Quota → fail-closed, accuracy-first

• Constraint: every over-admission has dollar cost; audit trail required
• Algorithm: token bucket with centralized Redis; stricter coordination
• Failure mode: fail-closed (503 or 429 with "service temporarily unavailable") + immediate alert
• Who pays for imperfection: finance/legal (over-admission means giving away paid resources)

Multi-Tenant Fairness → isolation-first

• Constraint: one tenant's traffic burst must not starve other tenants
• Algorithm: hierarchical token buckets — global cap → tenant cap → sub-tenant cap
• Failure mode: per-tenant circuit breaking; global capacity preserved
• Who pays for imperfection: product team (explaining why enterprise customer was throttled)

2.2 What the Interviewer Leaves Underspecified

Interviewers deliberately omit:

• Auth vs unauth traffic
• Client identity strength
• Hard vs soft limits
• Multi-region behavior
• Regulatory constraints

Staff engineers surface these. Senior engineers assume them away.

2.3 Precise Terminology

Rate-limiter interviews are ambiguous about where enforcement runs. Use precise terms:

3The Five Fault Lines

3.1 Fault Line 1: Protection vs Correctness

The tension: Strict correctness requires coordination that can become the bottleneck. Protection-first accepts drift but keeps the system alive.

L6 answer: "For abuse protection, I choose protection-first with bounded drift. I'd rather over-admit 10% than add 10ms to every request or cause a self-inflicted outage when Redis degrades. For billing/quota, I flip to correctness-first — the over-admission has direct cost. The key insight: abuse protection and billing cannot share the same rate limiter configuration because they have incompatible failure modes."

L7 answer: Reframes as risk governance — "Which layer enforces what?" CDN/WAF for coarse abuse, gateway for identity-aware limits, app for business invariants. Defines who signs off on fail-open/closed and what blast radius is acceptable.

3.2 Fault Line 2: Centralized vs Distributed State

The tension: Central state is easy to reason about but creates a dependency in the hot path. Distributed state is resilient but requires explicit coordination mechanisms.

L6 answer: "I pick token leasing for this design. Each gateway instance leases a chunk of tokens from Redis (say, 50 tokens for a 1,000 req/min limit with 10 gateways), spends them locally with zero coordination per request, then renews when the lease runs low. Redis QPS drops from request-per-lease-renewal to approximately 2 RPCs per gateway per 5 seconds — a 10–100× reduction. Failure mode: if a gateway crashes mid-lease, those leased tokens are stranded for the lease duration (5 seconds). For abuse protection, 5 seconds of stranded capacity is acceptable."

L7 answer: "Do we even need custom distributed coordination? Evaluate managed gateway throttling (Envoy RLS, AWS API Gateway throttling, Cloudflare Rate Limiting) before building custom. If custom is needed, select the coordination mechanism based on operational cost: token leasing for abuse protection at scale, key-owner routing for authenticated billing paths."

3.3 Fault Line 3: Latency vs Accuracy

The tension: Every millisecond added to the critical request path has compound effects at scale. At 1M req/sec, a 5ms Redis round-trip is the difference between a healthy gateway and a latency amplifier.

L6 answer: "The latency tax is the reason the local-first design exists. At 100K req/sec, every 1ms of rate-limiter overhead is 100 additional seconds of cumulative delay per second of traffic. I keep the hot path at ~0ms by checking local buckets first. Redis is for coordination, not for enforcement. The 5–10ms overhead only occurs on lease renewals (every 5–10 seconds per gateway instance) or on the first request from a new identity."

3.4 Fault Line 4: Fail-Open vs Fail-Closed

The tension: When the rate limiter's central store fails, you must choose between protecting availability (fail-open, risk abuse) and protecting resources (fail-closed, risk self-inflicted outage).

Guardrails for fail-open:

• Aggressive Redis timeout (5–10ms max — fail fast, don't let threads pile up)
• Conservative local fallback caps (set per-instance limit to 2× normal, not unlimited)
• Bypass-rate alerting: bypass_rate > 1% for 5 min → page on-call
• Circuit breaker on backend stress signals: if downstream error rate rises while in fail-open, tighten caps

L6 answer: "I make the fail-open/fail-closed decision before writing a single line of code, and I tie it to intent. For this abuse protection limiter: fail-open. The rate limiter should never become a kill switch for legitimate users. I add three guardrails: aggressive Redis timeout (5ms), conservative local fallback caps, and bypass-rate alerting. If the backend shows stress while in fail-open mode, the fallback caps tighten automatically. For billing: fail-closed with immediate on-call escalation — we can't give away paid resources."

L7 answer: Defines a governance model — who can flip fail-open/closed (change management approval required), what the emergency procedure is, what the kill-switch scope is (per-endpoint, per-tenant, or global), and what post-incident analysis is required.

3.5 Fault Line 5: Infra Ownership vs Team Autonomy

The tension: A central rate limiting service creates consistency but becomes a bottleneck. Per-service SDKs give teams flexibility but cause library hell and config drift.

L6 answer: "Enforce at the gateway/sidecar to avoid per-service SDK drift. Within 6 months of distributing a rate limiting SDK, I'd have 3–4 incompatible versions in production — one with a security bug the team hasn't updated, one on a deprecated version. A policy change requires coordinating 15 teams instead of one config update. The gateway enforcement model means: platform team owns infrastructure, product teams own policy intent via a self-service API. Policy changes deploy to the gateway — no service redeploy required."

L7 answer: Defines governance: who owns policy definition (product team, within platform-defined bounds), who approves exceptions (architecture review for limits above platform maximums), how staged rollouts work (shadow → warn → enforce), and how you prevent the platform team from becoming a bottleneck.

4Failure Modes & Operational Reality

4.1 Store Failures — Full Timeline

Scenario A: Redis becomes slow (most common)

t=0:     Redis p99 jumps from 2ms to 100ms
t=0–30s: Gateway worker threads pile up waiting for Redis responses
t=30s:   Gateway request queues fill — new requests start queuing
t=1min:  Gateway starts returning 503s to ALL traffic
t=2min:  "The rate limiter" has become a global outage
         (NOT a Redis outage — a rate limiter outage)

What breaks first: P99 latency at the gateway spikes, threads pile up, the rate limiter becomes a latency amplifier for every request regardless of rate limit status.

Bad reaction: "Increase Redis timeouts" — makes it worse, extends the window before circuit break fires.

Staff reaction: Aggressive Redis timeout (5ms) triggers circuit break → switch to local-only mode → bypass-rate metric fires alert → on-call is paged within 30 seconds. The gateway never queues on Redis.

Scenario B: Redis is completely down

Staff choice for abuse protection: Fail-open with aggressive local limits, bypass-rate alerting, and circuit breaker on backend stress. The rate limiter must not become a global kill switch.

4.2 Hot Key & Amplification — Why "Just Shard It" Doesn't Work

The critical insight: sharding helps for many different keys. It does NOT help for one single identity that generates 100K req/s — that identity always maps to the same shard regardless of how many shards exist.

Scenario A: Leaked API key used by botnet

• Symptom: one API key drives 50%+ of traffic; single shard CPU spikes
• Mitigation: (1) local deny-cache — reject locally with TTL 30–60s without hitting Redis, (2) revoke/rotate the key, (3) add CDN/WAF edge rules if stable IP/ASN signals
• Tradeoff: fast containment vs false positives if key is shared with legitimate partner

Scenario B: Legitimate tenant burst (partner batch job)

• Symptom: paying tenant looks like "hot key" but isn't malicious
• Mitigation: token leasing with adaptive lease sizing for higher-tier tenants; per-tenant reservation pool
• Tradeoff: fairness vs utilization

Scenario C: IP-based identity collapses (NAT/corporate proxy)

• Symptom: one IP = thousands of real users → false positives
• Mitigation: IP as coarse outer limiter only; shift to stronger identity (API key / user_id) for authenticated paths
• Tradeoff: better UX vs implementation complexity

Mitigations that actually work for hot keys:

1. Local aggregation: batch increments every 100ms instead of per-request — 100× Redis QPS reduction
2. Key-owner routing: single gateway instance owns enforcement for hot identity — no multi-writer races
3. Deny-cache: if identity is already 10× over limit, reject locally without hitting Redis at all
4. Key sharding + sum-on-read: split rl:{key}:{window} into rl:{key}:{window}:{shard_0..7} and sum at read time — distributes write load across shards

4.3 Data Integrity Failures

Clock Skew: Token refill depends on wall time. Drift causes over-refill or under-refill across nodes.

• Mitigations: cap refill deltas (never trust a >10s time jump), use monotonic clocks, use Redis server time inside Lua scripts
• Staff note: redis.call("TIME") inside the Lua script uses Redis server time, reducing clock-skew risk — adds ~0.1ms overhead but worth it for billing paths

Script Bugs: Atomic Lua operations silently wrong. Hardest failure mode to detect.

• Detection: integration tests covering boundary conditions, audit logging with remaining_tokens sampled 1%
• Recovery: script rollback (keep previous version deployable), counter reset procedure

4.4 Operational Reality Matrix

5Evaluation Rubric

5.1 Level-Based Signals

5.2 Strong Hire Signals

5.3 Lean No-Hire Signals

5.4 Common False Positives

• Knows Redis deeply: Deep Redis knowledge ≠ good system design. Implementation detail fluency without tradeoff reasoning.
• Draws complex diagrams: Complexity is not a Staff signal. Can they explain the organizational cost of each component?
• Mentions many algorithms: Breadth without depth is Senior, not Staff.
• "We'll use Envoy RLS": Naming a managed solution without explaining when custom is needed and when managed is sufficient.

6Interview Flow & Pivots

6.1 Typical 45-Minute Shape

6.2 Reading the Interviewer

6.3 What to Deliberately Skip

6.4 Follow-Up Questions to Expect

1. "How do you handle clock skew across gateway instances?"
2. "What if a single user generates 90% of traffic?"
3. "How do you test rate limiting logic in production?"
4. "What metrics would you monitor to detect a silent fail-open?"
5. "How do you handle a global rate limit across multiple regions?"
6. "What's your failure budget for this service?"

7Active Drills

Global capacity bucket (protects platform overall)
  └── Tenant A bucket (contracted: 500 req/s, burst: 800 req/s)
       └── Sub-buckets per user/endpoint within Tenant A
  └── Tenant B bucket (contracted: 200 req/s, burst: 300 req/s)
  └── Shared burst pool (available to all tenants when slack exists)

Deep Dive 1: Flash Sale Incident

Context: It's Black Friday. The rate limiter is returning 429s to legitimate users during a 10× traffic spike. The on-call engineer escalates to you.

Questions to Surface First:

• Is the backend actually unhealthy, or is the limiter rejecting traffic the backend could handle?
• Is this affecting all users or specific tenant tiers? Are VIP/enterprise customers hitting public limits?
• Was this traffic spike predicted? Was there a capacity planning exercise for Black Friday?
• What's the business cost per minute of rejecting legitimate buyers right now?

Staff Signals:

• Checks backend health before blaming the rate limiter
• Has pre-established emergency procedures, not ad-hoc judgment calls
• Identifies the organizational gap (capacity planning didn't include Black Friday) rather than just fixing the immediate symptom

Deep Dive 2: Silent Fail-Open

Context: You discover the rate limiter has been running in fail-open mode for 3 days because Redis was slow. No alerts fired. What went wrong, and how do you fix the system?

Questions to Surface First:

• How long has fail-open been active? Did abuse increase during this window — check login failure rates, API error rates, unusual traffic patterns.
• Is fail-open an intentional design choice, or did it happen by accident (no fallback configured)?
• What other services degrade silently when their dependencies are slow? Is this a systemic pattern?
• Who should have been alerted, and why wasn't existing monitoring sufficient?

Staff Signals:

• Treats this as a systemic observability contract failure, not a Redis monitoring gap
• Proposes time-bounded fail-open that auto-expires and requires explicit re-approval
• Audits all services with similar silent degradation modes

Deep Dive 3: Large Tenant Onboarding

Context: Sales just closed a deal 5× larger than your current biggest tenant. They go live in 2 weeks. Ensure the rate limiter handles them without affecting other tenants.

Questions to Surface First:

• What's this tenant's expected traffic pattern — steady-state vs bursty? What are their peak hours?
• If this tenant misbehaves, what's the blast radius to other tenants? Are counters shared with others on the same Redis shard?
• What SLA did Sales promise? Is it documented, or implicit? Who approved the capacity commitment?
• Do we have tenant isolation at the infrastructure level, or are all tenants sharing the same Redis shard?

Staff Signals:

• Reframes from handling load to blast radius isolation
• Uses shadow mode before hard enforcement for high-risk capacity changes
• Negotiates explicit SLA documentation with Sales before committing infrastructure

Deep Dive 4: Post-Mortem — Limiter Let Through an Attack

Context: Post-mortem shows a credential-stuffing attack got through the rate limiter for 45 minutes. 50K accounts were compromised. What do you present to leadership?

Questions to Surface First:

• Was rate limiting ever designed to be the primary defense against credential stuffing, or was it assumed?
• What other defense layers exist (WAF, CAPTCHA, anomaly detection)? Which ones fired, which didn't?
• How did the attacker evade detection — IP rotation, distributed botnet, credential replay from a breach database?
• What's the regulatory exposure? Do we need to notify affected users under GDPR/CCPA?

Defense-in-depth: rate limiting (Layer 2) cannot stop sophisticated attacks alone. Layers 3 and 4 catch what volume-based rules miss.

Staff Signals:

• Presents to leadership with architectural framing (rate limiting was never the right sole defense) rather than just fixing the specific gap
• Proposes layered defense-in-depth, not just tighter thresholds
• Defines ownership boundaries across Security and Platform teams as an organizational action item

Deep Dive 5: Multi-Region Expansion

Context: Your company is expanding to EU. The rate limiter is currently single-region. What's your recommendation?

Questions to Surface First:

• What's the primary use case for EU expansion — latency reduction, data residency compliance, or both?
• Which rate limiting intents need global accuracy (billing/quota) vs regional approximation (abuse protection)?
• Do rate limit counters contain PII or request metadata that falls under GDPR data residency requirements?
• What's the acceptable drift window — can a global abuser temporarily get 2× their limit across two regions?

Hybrid approach: abuse protection runs independently per region with async sync; billing/quota uses region-scoped quotas or tighter global coordination where accuracy matters.

Staff Signals:

• Separates abuse protection (tolerates drift) from billing/quota (needs accuracy) rather than applying one architecture to both
• Raises GDPR/data residency constraints on counter data — the non-technical dimension most candidates miss
• Plans a phased rollout rather than a big-bang migration

9Level Expectations Summary

After studying this playbook, you should be able to:

• Name the three intents and explain why they require incompatible failure modes
• Quantify the latency trap — Redis per-request at 5–10ms overhead at scale
• Design a local-first coordination model with a named mechanism (token leasing, key-owner routing, bounded reconciliation) and its specific failure mode
• Explain fail-open vs fail-closed by intent, with bypass-rate observability as a mandatory component
• Walk through the silent fail-open failure mode from detection to runbook
• Explain why hot keys are structurally different from high-cardinality traffic and propose mitigations
• Design a policy management system with a lifecycle (shadow → warn → canary → enforce) and organizational governance

The Bar for This Question

Mid-level (L4/E4): Understands rate limiting as a counter in Redis with a TTL. Can explain token bucket semantics. Knows rate limiting prevents abuse. Doesn't reason about distributed coordination, failure modes, or ownership.

Senior (L5/E5): Builds the baseline architecture quickly — gateway middleware + Redis counters with Lua atomicity. Reasons about the consistency-availability tradeoff. Has an opinion on local vs centralized counters and can explain the latency implications of a synchronous Redis call. Gets to fail-open vs fail-closed when prompted.

Staff+ (L6/E6+): Spends under 5 minutes on architecture and 30+ minutes on depth. Quantifies the latency trap and names the coordination mechanism (leasing, routing, or reconciliation) with its failure mode. Makes fail-open/fail-closed a proactive decision, not a response to the interviewer's question. Defines bypass-rate observability as mandatory. Reasons about policy management as an organizational problem. The interviewer should learn something from the answer.

10Staff Insiders: Controversial Opinions

10.1 "Exact Rate Limiting" Is a Myth

At scale, you are never enforcing the limit you think you're enforcing.

The Staff position: Stop pretending you're enforcing "exactly 1000 req/s." You're enforcing "approximately 1000 req/s ± ε." The Staff question is: what's your ε, and is it acceptable for your intent?

Why this matters in interviews: Candidates who claim exact enforcement without acknowledging drift reveal they haven't operated rate limiters at scale. The bar-raiser question: "What's the worst-case over-admission in your design, and who signed off on it?"

10.2 Abuse Protection and Billing Cannot Share an Algorithm

If you're using the same rate limiter for abuse protection and billing enforcement, one of them is wrong.

The Staff position: These are fundamentally different systems. Trying to serve both with "one rate limiter" leads to billing drift (abuse limiter too loose) or availability problems (billing limiter too strict for abuse). Stripe figured this out and separated them. Companies that conflate them eventually have an incident.

10.3 Global Fairness Dies at Scale — And That's OK

Many companies that claim "global rate limiting" are actually doing per-region rate limiting and calling it global. At true global scale, they've abandoned global fairness.

The dirty secret: At hyperscale, a user with a "1,000 req/min global limit" might actually get 1,000 × N (where N = number of regions) if they distribute traffic across regions.

The Staff position: Global fairness is a spectrum. The honest question is: "What's the blast radius of our approximation, and is that acceptable?" At hyperscale, the answer is almost always "yes — 2× regional burst is acceptable for abuse protection."

10.4 The Policy Management Problem Is Harder Than the Technical Problem

The rate limiter is a solved technical problem. The unsolved problem is getting 15 product teams to agree on rate limit policies, keep them updated, respond when limits are hit, and not override the platform team's guardrails when their customers complain.

In practice:

• Product teams set limits too high because they fear blocking their customers — the limits become theater
• Product teams set limits too low and then complain when customers are throttled
• Nobody updates limits when a product changes behavior significantly
• The platform team becomes a bottleneck for every policy change, or abdicates policy ownership entirely

The Staff position: Rate limiting's technical implementation is a 2-week project. Rate limiting's organizational governance is a 6-month project. Staff engineers work on the 6-month project after shipping the 2-week project. Self-service policy API with guardrails, policy lifecycle with shadow mode, and clear ownership boundaries are the deliverables that actually matter long-term.

t=0.99s: 100 requests → Allowed (window 1)
t=1.01s: 100 requests → Allowed (window 2 just started)
Result:  200 requests in ~20ms

Users can time requests to get 2× burst at every window boundary.

t=0.0s: 150 requests → 150 allowed (tokens: 200→50)
t=0.5s: 120 requests → refilled 50 tokens (50→100)
                     → 100 allowed, 20 rejected (429)
t=0.6s: 10 requests  → refilled ~10 tokens (0→10)
                     → 10 allowed (tokens: 0)

count ≈ (prev_window_count × overlap_fraction) + current_window_count

rate_limit:{identity}:{scope}:{window}

Examples:
  rate_limit:ip:203.0.113.42:/login:60s        # IP + endpoint + window
  rate_limit:apikey:abc123:/search:10s          # API key + endpoint + window
  rate_limit:user:u_789:/api/v1:1m              # User ID + API prefix + window
  rate_limit:tenant:t_42:/api/v1:1m             # Tenant ID for multi-tenant fairness

-- KEYS[1] = bucket key
-- ARGV: capacity, refill_rate_per_sec, now_ms, cost_tokens
-- Returns: {allowed (0/1), remaining_tokens, retry_after_ms}

local tokens, last_ms = ...
local elapsed_ms = now_ms - last_ms
local refilled = elapsed_ms * refill_rate_per_sec / 1000
tokens = math.min(capacity, tokens + refilled)

if tokens >= cost then
    tokens = tokens - cost
    -- store updated tokens, return allowed=1
else
    -- return allowed=0, retry_after_ms = ceil((cost - tokens) / refill_rate * 1000)
end

hash(api_key) % gateway_count → routes to consistent owner
Owner handles all enforcement locally, no coordination needed

if local_tokens < low_watermark:
    force_global_check(key)  # One-time Redis call, not per-request

# Periodic: every 500ms
send_usage_deltas(key, local_count_delta)
Redis reconciles global budget across all gateways

HTTP/1.1 429 Too Many Requests
Retry-After: 2
Content-Type: application/json

{"error": "rate_limit_exceeded", "retry_after_seconds": 2}

HTTP/1.1 200 OK
X-RateLimit-Remaining-Tokens: 42
X-RateLimit-Refill-Rate: 100
X-RateLimit-Burst-Capacity: 200

retry_after_seconds = ceil((cost - tokens_remaining) / refill_rate)

429 received → retry immediately → another 429 → retry immediately...
10,000 clients each retry 3 times → 30,000 requests in the next 100ms

# Three categories every request falls into:
rate_limit_allowed_total{identity, endpoint, tier}
rate_limit_rejected_total{identity, endpoint, tier, reason}
rate_limit_bypassed_total{identity, endpoint, reason}  ← SILENT FAIL-OPEN DETECTOR

# Infrastructure health:
rate_limit_check_latency_ms{path}  # local vs Redis path
redis_latency_p99{operation}       # lease renewal, counter check
redis_timeout_total{operation}     # circuit break trigger

# Policy health:
rate_limit_enforcement_rate_pct    # % of requests checked against Redis (vs local only)

Global capacity bucket (protects platform)
  └── Enterprise Tier bucket (protects other tenants)
       └── Tenant A bucket (contracted 1,000 req/s burst 1,500)
            └── Sub-buckets per user/endpoint within Tenant A
       └── Tenant B bucket (contracted 500 req/s burst 750)
  └── Pro Tier bucket
  └── Free Tier bucket

rate_limit_allowed_total{tenant}
rate_limit_rejected_total{tenant, reason}
request_latency_p99{tenant}  ← fairness failures appear here first