Design a Distributed Cache

How to Use This Playbook

Organized for interview use first, reference second. Read front-to-back once. Return to individual sections for targeted review.

The L5 vs L6 Contrast — Start Here

What This Interview Actually Tests

Distributed caching is not a "make it faster with Redis" question. This is a consistency and organizational ownership question that tests:

• Whether you reason about invalidation before discussing eviction
• Whether you can articulate what "stale" means for your specific use case and who decided
• Whether you understand the blast radius when the cache fails at 3× origin load
• Whether you know when caching makes things worse, not better

The Staff Positions

The Three Intents

Commit to one intent before designing anything.

What Interviewers Probe

Quick-Reference: The 30-Second Cheat Sheet

System Architecture Overview

Phase 1: Frame the Problem (2 minutes)

Before drawing any boxes, surface the three questions that change the entire architecture.

Say this:

• "Before I design anything: what's the read/write ratio? What staleness is acceptable per data type? And who owns the write path — because invalidation ownership is where every cache breaks."

Commit to intent and assumption:

• "I'll assume origin protection on a read-heavy workload — 95% reads, mixed data types with per-type staleness budgets. That's the hardest case: it requires both availability reasoning (cache failure) and consistency reasoning (multi-writer invalidation). I'll design for that."

Name the "when not to cache" check explicitly:

• "Quick check before adding any cache: write rate close to read rate? If so, invalidation churn may exceed savings. Consistency-critical data? Financial, auth, permissions bypass the cache entirely. Data already hot in the DB buffer pool? Adding a network hop would make it slower. None of those apply here, so caching is appropriate."

Phase 2: Core Entities (30 seconds)

State three first-class entities — not just GET and SET.

• CacheEntry — key, value, TTL, version/etag (version enables conditional invalidation)
• InvalidationEvent — key_pattern, source, timestamp, propagation_status (first-class, not an afterthought)
• CachePolicy — data_type, staleness_budget, eviction_strategy, origin_fallback behavior

Phase 3: The 3-Minute Architecture (3 minutes)

Walk four layers in under 3 minutes.

1. Read path (cache-aside). App checks L1 in-process cache first (microseconds, 10s TTL), then Redis L2 (milliseconds, 5min TTL), then PostgreSQL with request coalescing on miss. Request coalescing: if 100 concurrent requests miss the same key, only one hits the database. Others wait and receive the same result. Prevents thundering herd without additional infrastructure.

2. Write + invalidation. App writes to PostgreSQL. Synchronously DELs the Redis key — delete, not update, to avoid race conditions between concurrent writers. Publishes to Redis pub/sub channel for L1 eviction across all app instances. CDC consumer provides an async backstop: if the write path missed the explicit DEL, the database changelog catches it within ~1 second.

3. Failure path. Circuit breaker on origin. When Redis is unavailable, requests fall through to PostgreSQL via the circuit breaker. Decision by data type: user preferences → serve stale from L1 if available; financial data → fail explicit with clear error. Request coalescing prevents stampede on the database during cache failure.

4. Layered caching. L1 in-process (per instance, 10s TTL) → L2 Redis cluster (shared, TTL by data type) → CDN edge (semi-static content, 60s). Each layer has different staleness tolerance. Total worst-case staleness: L1 TTL + L2 TTL. Product signed off on that number for each data type.

Phase 4: Transition to Depth (15 seconds)

"The happy-path architecture is well-understood. What makes this Staff-level is the consistency and operational reasoning. Three areas: invalidation ownership — who's responsible when five teams write to the same entity; failure modes — thundering herd, hot key, cold start, and silent staleness bugs; and organizational ownership — who signs the staleness contract. Which do you want to go into?"

Phase 5: Deep Dives When Probed (5–15 minutes)

Probe A: "What happens when invalidation fails?"

"This is the silent failure mode. DB write succeeds. Invalidation fails (network blip, Redis timeout, async drop). No error, no alert. Users see stale data until TTL expires. Three layers of defense: (1) TTL as safety net — caps the worst-case window. (2) CDC backstop — database changelog publishes an invalidation event even if the write path missed it, arriving within ~1 second. (3) Correctness monitoring — sample a small percentage of reads, compare cached value against source of truth, alert on divergence. Most teams measure cache hit rate — a performance metric. Almost nobody measures cache correctness. That's where incidents hide."

Probe B: "Walk me through thundering herd."

"Popular cache key expires. 1,000 concurrent requests check the cache simultaneously. All miss. All hit the database. Database connection pool exhausts in seconds. Service degradation. This is the most common cache failure mode and almost always under-mitigated. Three defenses in order of preference: (1) Request coalescing (singleflight) — first miss acquires a lock, fetches from origin, populates cache. Other 999 wait for the result. Cost: added latency for waiters, but only one origin request. (2) Probabilistic early expiry — entries expire slightly before their TTL with some jitter, spreading refresh load over time instead of at one cliff-edge moment. (3) Background refresh — before expiry, a background process refreshes the entry so it never actually misses. Cost: always slightly stale. Coalescing is the default; probabilistic jitter is the enhancement."

Probe C: "The cache is down. What happens?"

"Distinguish by data type immediately. Financial data, auth tokens, permissions: fail explicit — circuit breaker returns a clear error, no serving of stale data. User preferences, product descriptions, feed content: serve stale from L1 if available; if not, fall through to PostgreSQL with request coalescing and circuit breaker. The circuit breaker on origin is non-negotiable — if Redis handles 90% of read load, a cold fallback means 10× the origin load. Without a circuit breaker, the origin collapses under the cache-failure load, turning a cache incident into a full service outage."

Probe D: "How do you handle hot keys?"

"One product goes viral. One Redis shard gets 100× normal traffic. Shard CPU hits 100%, latency spikes, shard fails. Every request for that key now misses everywhere. Three mitigations: (1) L1 in-process caching for predictable hot keys — absorbs traffic at the application tier with a short TTL. (2) Key replication — distribute the hot key across N shards with {key}:shard_id suffix; read operations choose a random shard. Memory cost: N copies. Invalidation complexity: N DEL operations. (3) Read replicas — route hot-key reads to Redis replicas. Eventual consistency: stale by replication lag. Detection: track per-key QPS metrics; alert when a single key exceeds a percentage of total shard QPS. Hot keys are usually predictable (homepage, trending, viral content) but occasionally surprise — the detection metrics matter."

Phase 6: Close with Constraints (30 seconds)

"Distributed caching is a consistency problem disguised as a performance optimization. The performance part is easy — add Redis, add TTL. The consistency part is the real work: who owns the invalidation contract when five services write to the same entity? Who signed off on 30 seconds of staleness for this data type? Who gets paged when the cache serves wrong data for three hours and nobody notices? Those organizational answers determine whether the cache runs itself or generates an incident every month."

1The Staff Lens

1.1 Why Caching Separates L5 from L6

Caching is the system design topic with the highest gap between apparent simplicity and operational complexity. Everyone has used Redis. Almost nobody has debugged a silent invalidation failure that served wrong data for six hours before anyone noticed.

The interviewer is testing whether you see caching as a consistency contract with organizational ownership — not just a performance optimization with a TTL setting.

1.2 The L5 vs L6 Contrast — Visual

1.3 The Key Insight

2Problem Framing & Intent

2.1 When NOT to Cache

Staff candidates win interviews by knowing when to push back. This is the highest-signal behavior.

2.2 The Three Intents

Intent 1: Origin Protection

• The origin can't handle all reads. Caching absorbs the majority of load.
• Failure mode: cache down = DB overload. Circuit breaker and request coalescing are non-negotiable.
• Key metric: origin RPS (not just hit rate). Origin RPS rising while hit rate holds means you have a new hot key.

Intent 2: Latency Reduction

• Reads must be sub-millisecond. Network hops to the DB are too slow.
• Multi-tier caching: L1 in-process for the hottest keys, L2 Redis for the warm tier.
• Staleness tolerance defines which data can be in L1 (shorter TTL) vs L2 (longer TTL).

Intent 3: Cost Optimization

• Expensive computation (ML inference, complex joins) is cached to avoid recomputation.
• Longer TTLs are acceptable — these results change infrequently.
• Invalidation on model update or data change; TTL as the outer bound.

2.3 What the Interviewer Leaves Underspecified

Interviewers deliberately omit:

• Read/write ratio
• Staleness tolerance per data type
• Whether consistency is more or less important than availability
• Number of write services (invalidation coordination complexity)
• Cache budget (memory)

Staff engineers surface these. Senior engineers assume them away.

2.4 Terminology Reference

3Fault Lines

3.1 Fault Line 1: Freshness vs Performance

The tension: Shorter TTLs mean fresher data but more origin load. Longer TTLs reduce origin load but guarantee staleness. The resolution: staleness is a business decision owned by Product, not a technical constant owned by Engineering.

L6 answer: Per-type staleness budget with explicit invalidation as primary mechanism and TTL as safety net. Classify data types: real-time (<10s) → inventory, pricing; near-real-time (30s–5min) → user profiles, catalog descriptions; batch (>5min) → aggregations, pre-computed feeds. Product signs off on each classification. Engineering sets the TTL to match the classification plus buffer.

3.2 Fault Line 2: Cache-Aside vs Write-Through vs Write-Behind

The tension: Write-through guarantees cache-DB consistency but couples write availability to cache availability. Cache-aside decouples availability but requires the application to own invalidation. Write-behind improves write performance but risks data loss if the cache fails before the async write completes.

L6 answer: Cache-aside for most cases, with explicit write-path invalidation as primary mechanism and CDC backstop for anything that gets missed. "Cache-aside means: if the cache is down, writes still succeed and reads fall through to the database. Write-through means: if the cache is down, writes fail. At the scale where caching matters, cache-aside's availability advantage outweighs write-through's consistency simplicity."

3.3 Fault Line 3: Availability vs Consistency on Cache Failure

The tension: When the cache fails, serving stale data maximizes availability but may serve incorrect data. Hitting the origin maximizes consistency but risks overloading it. Failing explicit is honest but degrades the user experience. The right answer depends on the data type — and it's a product decision.

L6 answer: Classify every data type before the incident, not during it. Document the failure behavior in the service's runbook. The distinction between "serve stale" and "fail explicit" is a product decision that engineering should confirm in writing before shipping a cache layer.

3.4 Fault Line 4: Local vs Distributed Cache

The tension: In-process (local) caches are microsecond-fast but create per-instance consistency issues — different app instances serve different data. Distributed caches (Redis) are consistent across instances but add network latency per request. Multi-tier uses both.

L6 answer: Multi-tier for high-traffic systems. L1 absorbs hot-key pressure — if one product gets viral traffic, all instances have it in L1 and the Redis shard never sees the load spike. L1 TTL must be short (5–10s) because there is no push invalidation to L1 by default — unless you implement a pub/sub channel for L1 eviction, which is the right answer for any data where 10-second staleness is unacceptable.

3.5 Fault Line 5: Proactive vs Reactive Invalidation

The tension: Proactive invalidation (DEL on write) gives immediate freshness but requires every write path to know about and correctly implement invalidation. Reactive invalidation (TTL expiry) is simple but guarantees staleness up to TTL. The Staff answer is hybrid with a formal backstop.

L6 answer: Hybrid with CDC backstop. On every write: synchronously DEL the cache key. TTL of 2× the acceptable staleness window as safety net. CDC consumer as backstop: tails the database changelog, publishes invalidation events for any row change regardless of which write path caused it. This means: even if a developer adds a new write path and forgets to add cache invalidation, the CDC consumer catches it within ~1 second.

4Failure Modes & Operational Reality

4.1 Thundering Herd (Cache Stampede)

What happens: A popular cache key expires. Hundreds of concurrent requests miss simultaneously. All hit the database. DB CPU spikes, connections exhaust, service degrades.

Timeline:

t=0s:    Cache entry for "homepage_data" expires (TTL)
t=0-1s:  1,000 concurrent requests → all miss
t=0-1s:  1,000 DB queries → DB CPU 100%
t=1-2s:  DB connection pool exhausted
t=2-5s:  Query latency spikes to 10s
t=5s+:   Cascading failure — homepage down

Mitigations in priority order:

Staff answer: Request coalescing is the default. One lock per key; first acquirer fetches from origin; others wait. Combined with probabilistic early expiry to spread refresh load over time rather than at one expiry cliff.

4.2 Hot Key Problem

What happens: One cache key receives 100× normal traffic (viral content, trending product, popular user). Single Redis shard CPU hits 100%. Shard latency spikes. Requests fail.

Timeline:

t=0:     Content goes viral
t=1min:  One key: 10,000 req/s → single Redis shard overloaded
t=2min:  Shard CPU at 100%, latency spikes
t=3min:  Shard timeouts → fallback to DB
t=4min:  DB overwhelmed by thundering herd from failed shard

Mitigations:

Detection: Per-key QPS metrics. Alert when a single key exceeds X% of total shard QPS. Hot keys for predictable events (Black Friday homepage, scheduled content drops) should be pre-populated in L1 before traffic arrives.

4.3 Cold Start (Cache Miss Storm)

What happens: Redis cluster restarts (maintenance or failure). 100% miss rate. All requests hit the database. Service degrades within seconds.

Timeline:

t=0:     Redis cluster unavailable
t=0:     100% cache miss rate
t=0-30s: All reads hit PostgreSQL
t=30s:   DB connection pool approaches capacity
t=1min:  Service degradation or outage

Mitigations:

Staff answer: Warming is deployment-critical, not a background task. Traffic does not shift until the new cache reaches a hit-rate threshold (e.g., 80%). Warming duration is tracked as a trend metric — alert if it grows >50% over baseline (dataset growth outpacing the warming job).

4.4 Silent Staleness (The Invisible Failure)

What happens: Cache invalidation fails silently. Users see stale data. No errors, no alerts — just wrong answers.

Timeline:

t=0:     User updates email in UI
t=0:     DB write succeeds
t=0:     Invalidation call silently times out
t=0-5min: User sees old email in every view
t=5min:  TTL expires, correct data served

Why it's dangerous: No error rate. No latency spike. No alert fires. The only signal is a user complaint — or worse, nothing.

Mitigations:

• CDC backstop catches missed invalidations within ~1 second
• Correctness monitoring: sample 0.1% of reads, compare cached value to source of truth, alert on divergence
• Version stamps: make stale entries unreachable, not just wrong
• Short TTL fallback: cap the worst-case window

4.5 Operational Reality Matrix

5Evaluation Rubric

5.1 Level-Based Signals

5.2 Strong Hire Signals

5.3 Lean No-Hire Signals

5.4 Common False Positives

• Knows Redis data structures deeply ≠ good cache design. Sorted sets and HyperLogLog knowledge is not the bar.
• Multi-tier diagram ≠ multi-tier design. Drawing L1 + L2 + CDN without explaining aggregate staleness, L1 invalidation via pub/sub, and the failure mode of each tier is architectural decoration.
• Mentions CDC ≠ understands CDC. "Use CDC for invalidation" without explaining the lag, the infrastructure complexity, and when it's appropriate signals pattern-matching rather than operational experience.

6Interview Flow & Pivots

6.1 Typical 45-Minute Shape

6.2 How Interviewers Pivot

6.3 Follow-Up Questions to Expect

1. "What happens when the cache is completely cold?"
2. "How do you handle thundering herd when a popular key expires?"
3. "What's your invalidation strategy on write? What if the invalidation fails?"
4. "How do you detect that your cache is serving stale data without anyone noticing?"
5. "When would you NOT cache this data?"
6. "Your cache costs tripled last quarter. How do you investigate?"

7Active Drills

Deep Dive 1: Black Friday Cache Failure

Context: It's Black Friday. Your Redis cluster shows 50% higher latency than normal and cache hit rate has dropped from 95% to 70%. On-call escalates to you.

Questions to Surface First:

• Is this a cache problem or an origin problem? A 25% hit rate drop on Black Friday could mean 5× increase in database load — the database may be the real patient.
• What caused the hit rate drop — hot keys from a viral deal, memory pressure causing evictions, or traffic exceeding capacity plans?
• Were circuit breakers and fallbacks configured in advance, or are they being improvised during the incident?
• Was Black Friday traffic included in capacity planning? If not, this is a process failure, not a technical one.

Deep Dive 2: Stale Payment Data Incident

Context: A customer reports that after updating their payment method, the old card kept getting charged for 3 hours. Investigation shows the payment cache had a 3-hour TTL with no write-through invalidation.

Questions to Surface First:

• What category is this data — financial, PII, preferences? Different categories have different staleness tolerances, and financial data should have been classified as "never stale."
• Was the 3-hour TTL an intentional design choice, or a default that nobody reviewed?
• What other sensitive data (billing, auth tokens, permissions) might have similarly misconfigured cache settings?
• Who approved caching payment method data at all?

Deep Dive 3: Cache Warming Gone Wrong

Context: During deployment, the cache warming job ran for 2 hours instead of the expected 10 minutes. The old cache was decommissioned on schedule. The site was slow for 2 hours.

Questions to Surface First:

• Why was the old cache decommissioned before warming completed? Was there a readiness gate, or was the decommission time-based?
• What caused warming to take 12× longer — dataset growth, degraded query performance, or insufficient parallelism?
• Is warming time tracked as a capacity trend metric? Does anyone monitor it?
• Should cache warming block deployment traffic shift?

Deep Dive 4: Multi-Tier Intermittent Stale Data

Context: Users report intermittent stale data — sometimes fresh, sometimes stale. You have L1 (10s TTL) and L2 (5min TTL) caching. L2 is invalidated on write; L1 is not.

Questions to Surface First:

• What's the worst-case staleness window? With L1 at 10s and L2 at 5min, data can be stale for up to 5min 10s in the worst case.
• Is L1 invalidation wired to the write path, or does it rely solely on TTL expiration?
• Is the intermittent pattern because different app instances have different L1 state?
• Has product approved "up to 10 seconds stale from L1" explicitly?

Deep Dive 5: Cache Cost Tripled

Context: Finance reports Redis costs tripled in 6 months. Leadership asks for a 40% reduction without performance impact.

Questions to Surface First:

• Why did costs triple — organic traffic growth, a new service caching aggressively without review, or dataset bloat from missing eviction policies?
• What percentage of cached keys are accessed more than once before eviction? (Single-use cached data is pure waste.)
• What's the hit rate by key pattern? Are there entire categories with no measurable benefit?
• Is the 40% target based on analysis, or is it an arbitrary Finance number?

9Level Expectations Summary

After studying this playbook, you should be able to:

• Ask "what's the staleness budget and who signed off on it?" before drawing any boxes
• Design explicit write-path invalidation + TTL safety net + CDC backstop
• Explain thundering herd and design request coalescing with probabilistic early expiry
• Classify data types into failure behaviors (serve stale / fall through to origin / fail explicit)
• Calculate worst-case aggregate staleness for multi-tier caching
• Know when to push back on adding a cache entirely

The Bar for This Question

Mid-level (L4/E4): Implements cache-aside with Redis, sets reasonable TTLs, explains the basic read path. Understands cache hits and misses and why caching improves latency. Can describe LRU eviction.

Senior (L5/E5): Quickly establishes the caching pattern based on access patterns and spends time on real problems: cache invalidation strategy (TTL vs event-driven), thundering herd on cache miss, cache key design and hit rate, and the staleness contract. Can quantify: "We cache product catalog with a 5-minute TTL because the business accepts 5 minutes of stale pricing in exchange for 10× lower DB load."

Staff+ (L6/E6+): Dispatches the baseline architecture in 5 minutes and spends 25+ minutes on operational depth: multi-tier caching with aggregate staleness arithmetic, cache warming strategies for cold starts, the organizational question of who owns the staleness contract (product signs off on user-facing staleness, engineering provides the mechanism), failure mode analysis classified by data type, correctness monitoring (not just hit rate), hot key detection and mitigation, and when NOT to cache. The interviewer should see you treat caching as a consistency contract with organizational ownership — not a performance optimization with a TTL value.

10Staff Insiders: Controversial Opinions

10.1 "Most Stale Data Incidents Are Never Detected"

Your cache is probably serving stale data right now. You just don't know it. No errors, no alerts, no latency spike — just confidently wrong answers. Stale data is the only failure mode that generates no technical signal. The only signals are user complaints (if users notice) or business metrics (if the data is business-critical). Most teams measure cache hit rate. Almost nobody measures cache correctness. If you can't tell me the last time you validated cached values against source of truth, your cache correctness is an assumption, not a guarantee.

10.2 "High Hit Rate Can Mask Correctness Bugs"

99% hit rate might mean you're serving confidently wrong answers 99% of the time. A pricing bug writes an incorrect value to the database. That incorrect value gets cached. For the next 5 minutes (TTL), 99% of reads hit the cache — and all of them get the wrong price. Without caching, the bug affects 1% of reads (direct DB reads); some users notice immediately. With caching, the bug affects 99% of reads for the full TTL duration. The cache turned a transient bug into a widespread incident. Hit rate is a performance metric. Correctness is the safety metric.

10.3 "Removing a Cache Is Often the Right Fix"

Signs you should delete the cache: hit rate below 50%, write rate approaching read rate (invalidation churn), multiple incidents traced to staleness, origin can handle the load without it, or nobody can explain the staleness contract. Teams don't remove caches because "we already built it" and "it must be helping somehow." The Staff engineer asks: "What if we just didn't cache this?" This requires courage — removing infrastructure is politically harder than adding it. But the most resilient cache is the one that doesn't exist.

10.4 "Cache Invalidation Is an Ownership Problem, Not a Technical One"

"Cache invalidation is hard" is a cop-out. It's hard because nobody owns it. Invalidation fails because: the writer doesn't know about the cache, there are multiple caches (one forgot), nobody designed for concurrent write race conditions, or the cache contract was never documented. The technical fixes (CDC, version stamps, pub/sub) exist and work. The organizational fix is harder: every write path must have a named owner of its cache contract. Without that, you'll add CDC, and then a new write path will appear that bypasses the CDC pipeline, and the bug returns.

10.5 "TTL Was Too Long Is Never the Root Cause"

When stale data causes an incident, "reduce TTL" is the wrong fix. It treats the symptom (staleness duration) not the cause (invalidation not firing). Shorter TTL means more origin load — you pay in performance to paper over an ownership gap. The real root causes: writer didn't know to invalidate, invalidation code had a bug, invalidation was async and lost, nobody owned the cache contract. Every "TTL was too long" incident is actually an invalidation ownership failure. Fix the ownership, not the number.

if (current_time - creation_time) > (TTL - random(0, jitter)):
    expire()

memory_needed = working_set_size × overhead_factor
working_set_size = num_keys × avg_value_size
overhead_factor = 1.5–2× (Redis data structures, fragmentation)

Cache size: 10% of working set → Hit rate: ~70%
Cache size: 20% of working set → Hit rate: ~85%
Cache size: 50% of working set → Hit rate: ~95%
Cache size: 100% of working set → Hit rate: ~99%