Design a Distributed Cache

How to Use This Playbook

This playbook supports three reading modes:

What This Interview Actually Tests

Caching is not a "make it faster" question. Everyone knows Redis.

This is a consistency and operational ownership question that tests:

• Whether you understand why caching introduces complexity, not just speed
• Whether you reason about invalidation before discussing eviction
• Whether you can articulate what "stale" means for your use case
• Whether you understand the blast radius when the cache fails

The key insight: Caching is a consistency problem disguised as a performance optimization. Staff engineers reason about who pays for staleness and who owns the invalidation contract.

The L5 vs L6 Contrast (Memorize This)

The Three Caching Intents (Pick One and Commit)

The Five Fault Lines (The Core of This Interview)

1. Freshness vs Performance — Shorter TTLs mean fresher data but more origin load. Who decides the staleness budget?

2. Cache-Aside vs Read-Through vs Write-Through — Where does the invalidation logic live? Who owns it?

3. Availability vs Consistency — When the cache is down, do we serve stale data, hit origin, or fail?

4. Local vs Distributed — In-process cache (fast, inconsistent) vs shared cache (slower, consistent)?

5. Proactive vs Reactive Invalidation — Push invalidation on write, or let TTL expire? Who coordinates?

Each fault line has a tradeoff matrix with explicit "who pays" analysis. See §3.

Default Staff Positions (Unless Proven Otherwise)

Quick Reference: What Interviewers Probe

Jump to Practice

→ Active Drills (§7) — 10 practice prompts with expected answer shapes

System Architecture Overview

Interview Walkthrough: How to Present This in 45 Minutes

The HelloInterview-style guides walk you through each step at tutorial pace. That's fine for Senior candidates. At Staff level, the basics should take 10-12 minutes — fast enough that you spend the remaining 30+ minutes on the invalidation, failure, and consistency questions that actually determine your level.

The six phases below add up to 45 minutes. The ratios matter: phases 1-4 are deliberately compressed so phase 5 gets the lion's share of time. If you're spending more than 12 minutes before the transition to depth, you're pacing like an L5.

Phase 1: Requirements & Framing (2-3 minutes)

State functional requirements in 30 seconds — don't enumerate, state the category:

• "We need a distributed caching layer to reduce database load and serve repeated reads at sub-millisecond latency."

That's it. Don't list every data type or cache operation.

Invest time on non-functional requirements (this is the Staff move):

• "What's the staleness budget? Product needs to define acceptable staleness per data type — prices need <30s, product descriptions can tolerate 5 minutes, user sessions need zero staleness."
• Clarify: read-to-write ratio (100:1 justifies caching, 2:1 probably doesn't), dataset size (does it fit in memory?), consistency model (eventual vs strong)
• "I'll assume a read-heavy workload with per-type staleness budgets, because that's the most common production scenario and forces the hardest invalidation decisions."

Phase 2: Core Entities & API (1-2 minutes)

State entities quickly (30 seconds):

• CacheEntry — key, value, TTL, version/etag (the version enables conditional invalidation)
• InvalidationEvent — key_pattern, source, timestamp, propagation_status (first-class entity, not an afterthought)
• CachePolicy — data_type, staleness_budget, eviction_strategy, origin_fallback behavior

API (1 minute) — transparent cache-aside in the application layer, not a separate API:

get(key) → HIT(value, age) | MISS
set(key, value, ttl, invalidation_policy) → OK
invalidate(key_or_pattern, reason) → OK

The invalidation path is the one that matters:

on_write(entity) → invalidate(cache_key(entity), "source_write")

Phase 3: High-Level Architecture (5-7 minutes)

Draw the core cache-aside flow on the whiteboard:

Walk the interviewer through the four data flows (reference the full System Architecture diagram above for the complete multi-layer picture):

1. Read path → App checks Redis first; on miss, reads from PostgreSQL, populates cache with data-type-specific TTL
2. Write path → App writes to PostgreSQL, then invalidates the cache key (delete, not update — avoids race conditions)
3. Invalidation propagation → For critical data, CDC (change data capture) publishes invalidation events as a backstop — so even if the application forgets to invalidate, the database change stream catches it
4. Failure path → When Redis is unavailable, requests fall through to PostgreSQL with circuit breaker protection and request coalescing (singleflight) to prevent thundering herd

Scripted walkthrough: "Read path: the app server checks Redis first — cache-aside. On miss, reads from PostgreSQL, populates cache with TTL. We use request coalescing so if 100 concurrent requests miss on the same key, only one hits the database. Write path: app writes to PostgreSQL, then invalidates the cache key. CDC publishes invalidation events as a backstop."

Key points to hit on the whiteboard:

1. Cache-aside pattern — application controls both read and write paths (not write-through, which couples cache to write latency)
2. Redis Cluster with hash slots — 6 shards for horizontal scaling, consistent hashing for key distribution
3. Request coalescing — singleflight pattern prevents thundering herd on cache miss
4. Write-path invalidation — delete on write, not update; avoids race conditions between concurrent writers
5. CDN as first cache layer — browser cache → CDN edge → Redis → PostgreSQL; four-layer hierarchy

Then immediately flag the key tension: "This works for the happy path. The interesting questions are: what happens when Redis goes down and 100% of traffic hits PostgreSQL? Who owns the invalidation contract when 5 different services write to the same entity? And how do you detect that your cache is serving stale data without anyone noticing?"

Phase 4: Transition to Depth (1 minute)

At this point you have a correct, simple architecture on the board. Now you pivot:

"The basic architecture is well-understood — cache-aside with Redis and TTL-based expiry. What makes this Staff-level is the consistency and operational reasoning. Let me dive into three areas: (1) invalidation strategy and who owns it, (2) failure modes when the cache layer goes down, (3) how to detect and measure staleness in production."

Then offer the interviewer a choice:

"I can go deep on any of these. Which is most interesting to you?"

If the interviewer doesn't have a preference, lead with invalidation strategy — it's the most universally asked and the most misunderstood.

Phase 5: Deep Dives (25-30 minutes)

The interviewer will steer, but be prepared to go deep on any of these. For each, follow the Staff pattern: state the tradeoff → pick a position → quantify the cost → explain who absorbs that cost.

Fault Line 1: Freshness vs performance — the staleness budget (5-7 min)

Open with the business framing:

"Every cache entry has a staleness budget — the maximum age the business will tolerate. For product prices, that's <30 seconds (stale prices lose money). For product descriptions, 5 minutes is fine (nobody cares if a typo fix takes 5 minutes to propagate). For user sessions, it's zero (stale session = security vulnerability)."

Go deeper — walk through the TTL decision framework:

1. Classify data types by staleness tolerance: real-time (<10s), near-real-time (30s-5min), eventual (>5min)
2. For real-time data: TTL is a safety net, not the primary mechanism. Use event-driven invalidation (CDC or explicit delete-on-write)
3. For near-real-time: TTL alone is sufficient. Set TTL = staleness_budget × 0.8 (leave 20% margin for clock skew)
4. For eventual: Long TTL (hours/days) with background refresh. These entries are the highest-value cache entries — they offload the most database reads

The Staff follow-up: "The dangerous case is when someone sets a 24-hour TTL on price data because 'it rarely changes.' It doesn't change — until it does, and then customers see stale prices for 24 hours. That's why TTL ownership should be in the product spec, not the code."

Cross-reference §3.1 Freshness vs Performance for the full analysis.

Fault Line 2: Cache failure and thundering herd (5-7 min)

"When Redis goes down, every request becomes a cache miss. If you have a 95% hit rate and 10K requests/second, that means 9,500 requests/second that were hitting cache now hit PostgreSQL directly. Your database is provisioned for 500 requests/second. It dies."

Name the mitigations in order of priority:

1. Request coalescing (singleflight) — if 100 concurrent requests miss on key X, only one hits the database; the other 99 wait for the result. This alone handles 80% of thundering herd scenarios
2. Circuit breaker on the database — if database latency exceeds threshold, reject new requests with a fallback (stale data from local cache, degraded response, or 503)
3. Stale-while-revalidate — serve the expired cache entry while refreshing in the background. The client gets slightly stale data instead of a slow or failed response
4. Local in-process cache (L1) — small LRU cache in each app server (1000 hot keys). Survives Redis failures with degraded consistency

Quantify the recovery: "With singleflight + circuit breaker, a Redis outage degrades latency from 2ms to 50ms (database reads) but doesn't cascade. Without protection, the database dies within 30 seconds and recovery takes 5-10 minutes because the connection pool is exhausted."

Fault Line 3: Invalidation strategy — proactive vs reactive (5-7 min)

"There are three invalidation approaches: (a) TTL-only (reactive — you wait for expiry), (b) explicit invalidation on write (proactive — you delete when data changes), (c) CDC-driven invalidation (proactive + reliable — the database change stream triggers invalidation). I'd use a combination: TTL as a safety net + explicit invalidation for the write path + CDC as a backstop."

The organizational problem: "With 5 microservices writing to the products table, who is responsible for invalidating the product cache? If the price service updates the price but doesn't invalidate the cache, the product service serves stale prices. That's why CDC is the backstop — it doesn't depend on every service remembering to invalidate."

Cache-aside vs write-through vs read-through (3-5 min)

"Cache-aside gives the application full control — it's the most common pattern and the most debuggable. Write-through couples your write latency to cache latency (now every write is DB + cache round trip). Read-through hides the caching logic in a library but makes debugging harder — when data is stale, you can't tell if the cache missed, the TTL is wrong, or the invalidation failed."

Pick a position: "I default to cache-aside for most systems. Write-through only when the write volume is low and you need guaranteed cache warmth (e.g., configuration data). Read-through only when you want the cache to be the primary interface and can accept the debugging cost."

Operational maturity: measuring staleness in production (3-5 min)

"How do you know your cache is serving stale data? You can't just check TTLs — you need to measure actual staleness. The approach: periodically sample cache entries, compare their version/etag against the database source of truth, and report the staleness distribution."

Three metrics that matter:

1. Hit rate per data type — if product prices have a 99.9% hit rate with a 30s TTL, you're serving a lot of cached prices. Is that safe?
2. Staleness distribution — p50/p95/p99 age of cache entries at read time. If p99 staleness exceeds the business SLA, your TTL or invalidation is broken
3. Invalidation propagation delay — time between database write and cache invalidation completion. If this exceeds 5 seconds, your "real-time" invalidation isn't real-time

Phase 6: Wrap-Up (2-3 minutes)

Summarize the key tradeoff — don't just restate your architecture, synthesize the insight:

"Distributed caching is a staleness management problem, not a performance optimization. The Staff-level challenge is: who defines the staleness budget, who owns the invalidation contract, and how do you detect when the contract is violated? The architecture is straightforward — cache-aside with Redis, TTL as safety net, CDC for proactive invalidation. The hard problem is organizational: making sure every write path has a corresponding invalidation path, and measuring staleness in production rather than assuming TTLs are correct."

If time permits, add the counterintuitive insight:

"Sometimes the right answer is to remove the cache. If your cache hit rate is 30%, you're paying for Redis infrastructure to serve 30% of reads while adding invalidation complexity for 100% of writes. At that point, scale the database instead. Caching is not free — it's a complexity trade for latency. Only make that trade when the hit rate justifies it."

Common Timing Mistakes

Reading the Interviewer

What to Deliberately Skip

These topics are traps. L5 candidates spend time on them. Staff candidates name them, dismiss them, and redirect to what matters.

The pattern: acknowledge you know it, state your position in one sentence, redirect to the interesting problem — invalidation, staleness, and organizational ownership.

→ Continue to The Five Fault Lines (§3) for the Staff-grade tradeoff reasoning.

11. The Staff Lens

1.1 Why This Problem Exists in Staff Interviews

This is NOT a "speed up reads" question. Everyone knows how to add Redis.

This is a Consistency & Operational Ownership question that tests:

• Whether you understand that caching trades correctness for performance
• Whether you reason about invalidation contracts, not just TTLs
• Whether you can articulate failure scenarios and their blast radius
• Whether you understand the operational burden of cache infrastructure

1.2 The L5 vs L6 Contrast

Behavior 1: First move (understand the access pattern)

Behavior 2: Invalidation strategy (TTL is not a strategy)

Behavior 3: Failure handling (replicas don't answer thundering herd)

Behavior 4: Consistency model (know when caching hurts)

Behavior 5: Ownership (who warms, who monitors)

1.3 The Key Insight

22. Problem Framing & Intent

2.1 The Three Intents

Before drawing any boxes, ask Why? The caching strategy changes entirely based on intent:

This sentence alone separates L5 from L6.

2.2 What's Intentionally Underspecified

The interviewer deliberately avoids specifying:

• Read/write ratio
• Data size and cardinality
• Staleness tolerance
• Cache budget (memory cost)
• Multi-region requirements

Staff engineers surface these unknowns. Senior engineers assume them away.

2.3 How to Open (The First 2 Minutes)

1. Ask 1-2 clarifying questions about access pattern and staleness
2. State your assumption explicitly
3. Outline your plan: access pattern → caching strategy → invalidation → failure modes → observability

Example opening:

2.4 Terminology (Use Precise Words)

Caching interviews are ambiguous about where and how caching happens. Use precise terms:

2.5 When NOT to Cache (Staff Candidates Say No)

Staff candidates win interviews by knowing when to not cache. This is the highest-signal behavior.

Do NOT cache when:

Bar-Raiser Follow-up: "When would you tell the team NOT to cache this data?"

Expected answer: "If the write rate is close to the read rate, invalidation will dominate. If the data is consistency-critical and staleness causes bugs or compliance issues, caching is the wrong tool."

33. The Five Fault Lines

This section contains the Staff-grade tradeoff reasoning. Each fault line includes:

• A tradeoff matrix
• Explicit "who pays" analysis
• L6 vs L7 calibration
• Bar-raiser follow-up questions

3.1 Fault Line 1: Freshness vs Performance

The tradeoff: Every second of TTL is a second of potential staleness. But shorter TTLs mean more origin traffic.

L6 (Staff) answer: Ties staleness budget to business requirements. "For user profile data, 30 seconds is acceptable — users don't expect instant updates. For inventory counts, we need tighter consistency, so we use write-through invalidation with a short TTL fallback."

L7 (Principal) answer: Establishes org-wide staleness SLAs by data class. "We have three data tiers: real-time (no caching or <1s), near-real-time (sub-minute), and batch (hour+). Each tier has a standard pattern and observability."

3.2 Fault Line 2: Cache-Aside vs Read-Through vs Write-Through

L6 (Staff) answer: Chooses cache-aside for most cases because it decouples cache availability from read availability. "If the cache is down, we fall back to the DB — slower, but not broken. Write-through couples us too tightly."

L7 (Principal) answer: Evaluates pattern choice based on failure modes and organizational capability. "Cache-aside requires every team to implement invalidation correctly. If we don't have that discipline, read-through with a managed cache might be safer despite the coupling."

3.3 Fault Line 3: Availability vs Consistency

Decision Framework:

L6 (Staff) answer: Classifies data by consistency requirements. "User preferences can serve stale — users won't notice a 30-second delay. But inventory must hit origin on cache miss because overselling is worse than latency."

L7 (Principal) answer: Defines consistency tiers as organizational policy with standard patterns and observability for each tier.

Ownership note: In practice, availability-first choices (serve stale) shift cost to Users (stale data experience) and Support (complaint volume). Consistency-first choices (fail or hit origin) shift cost to Engineering (circuit breaker complexity) and Infra (origin capacity to handle cache-miss load).

→ For the complete decision framework, see →Degraded Mode Framework — applies to cache failures, origin protection, and graceful degradation.

3.4 Fault Line 4: Local Cache vs Distributed Cache

The tradeoff: Local caches are fast but create consistency issues across instances. Distributed caches are consistent but add network latency.

L6 (Staff) answer: Uses multi-tier for high-traffic data. "L1 in-process cache with 10-second TTL absorbs repeat requests within a single instance. L2 Redis handles cross-instance consistency. L1 staleness is bounded by its short TTL."

L7 (Principal) answer: Standardizes multi-tier patterns across the org with clear guidance on when to use each tier and how to reason about aggregate staleness (L1 TTL + L2 TTL).

3.5 Fault Line 5: Proactive vs Reactive Invalidation

L6 (Staff) answer: Uses hybrid — proactive invalidation on write with TTL as safety net. "On user update, we delete the cache key. TTL handles cases where invalidation fails or the write path changes."

L7 (Principal) answer: Implements change data capture (CDC) for invalidation at scale. "Rather than coupling invalidation to every write path, we tail the DB changelog and invalidate asynchronously. This decouples writers from cache knowledge."

→ For invalidation coordination patterns, see →Distributed Coordination Framework.

44. Failure Modes & Degradation

→ This section applies the →Degraded Mode Framework. Review it if you need the full availability vs consistency decision tree.

4.1 Thundering Herd (Cache Stampede)

The most common cache failure mode. When a popular cache entry expires or the cache restarts, many requests simultaneously miss the cache and hit the origin.

Scenario: Popular Cache Key Expires

Timeline:

t=0:     Cache entry for "homepage_data" expires (TTL)
t=0-1s:  1000 concurrent requests hit cache → all miss
t=0-1s:  1000 requests hit database simultaneously
t=1-2s:  Database CPU spikes to 100%, queries slow
t=2-5s:  Database connection pool exhausted
t=5s+:   Cascading failures, homepage down

What breaks first: Database connection pool, then query latency, then availability.

Mitigations:

Staff answer:

4.2 Hot Key Problem

Similar to rate limiting hot keys. One cache key receives disproportionate traffic, overwhelming a single cache shard.

Scenario: Viral Content

Timeline:

t=0:     Content goes viral
t=1min:  Traffic to one key grows 100x
t=2min:  Single Redis shard CPU at 100%
t=3min:  Shard latency spikes, timeouts begin
t=5min:  Shard becomes unavailable
t=5min+: All requests for hot key fail

Mitigations:

Staff answer:

4.3 Cold Start (Cache Miss Storm)

Scenario: Cache Cluster Restart

Timeline:

t=0:     Redis cluster restarts (maintenance, failure)
t=0:     100% cache miss rate
t=0-1m:  All reads hit database
t=1m:    Database connection pool exhausted
t=2m:    Service degradation or outage

Why "just restart" doesn't work: If your cache handles 90% of read load, a cold cache means 10x the origin load overnight.

Mitigations:

Staff answer:

4.4 Cache Inconsistency (Stale Data Bugs)

Scenario: Failed Invalidation

Timeline:

t=0:     User updates their email
t=0:     DB write succeeds
t=0:     Cache invalidation fails (network blip)
t=0-5m:  User sees old email in UI
t=5m:    TTL expires, fresh data served

What makes this insidious: Silent failure. No errors, no alerts. Just wrong data.

Mitigations:

Staff answer:

4.5 Operational Reality Matrix

55. Evaluation Rubric

5.1 Level-Based Signals

5.2 Strong Hire Signals

5.3 Lean No Hire Signals

5.4 Common False Positives

• Knows Redis deeply: Deep Redis knowledge ≠ good cache design
• Mentions all patterns: Breadth without depth is Senior, not Staff
• Complex diagrams: Multi-tier diagrams without invalidation reasoning

66. Interview Flow & Pivots

6.1 Typical 45-Minute Structure

6.2 How Interviewers Pivot

6.3 What Silence Means

• After tradeoff question: Interviewer wants you to reason aloud
• After "what about consistency?": You're missing staleness reasoning
• After definitive answer: They may disagree or want nuance

6.4 Follow-Up Questions to Expect

1. "What happens when the cache is cold?"
2. "How do you handle thundering herd?"
3. "What's your invalidation strategy on write?"
4. "How do you detect stale data bugs?"
5. "What's the staleness budget for this data?"
6. "When would you NOT cache this data?"

77. Active Drills

Practice these scenarios to internalize Staff-level thinking. Try answering before revealing the Staff approach.

→ For the complete framework, see →Build vs Buy Framework.

These scenarios test Staff-level operational thinking. Unlike drills (which test interview responses), deep dives test ownership reasoning.

Deep Dive 1: Black Friday Cache Failure

Deep Dive 2: Stale Data Incident

Deep Dive 3: Cache Warming Gone Wrong

Deep Dive 4: Multi-Tier Cache Debugging

Deep Dive 5: Cache Cost Optimization

99. Level Expectations Summary

What gets you each level in a caching interview:

What Separates Each Level

Quick Self-Check

Before your interview, verify you can answer:

• What's the read/write ratio threshold where caching stops helping?
• What's your invalidation strategy, and what's the TTL fallback?
• How do you handle thundering herd on cache miss?
• What's the staleness budget, and who signed off on it?
• When would you NOT cache this data?

The Bar for This Question

Mid-level (L4/E4): You should be able to implement cache-aside with Redis, set reasonable TTLs, and explain the basic read path (check cache → miss → query DB → populate cache → return). You can describe cache hits and misses and why caching improves latency. Understanding cache eviction policies (LRU) or invalidation challenges would be a bonus but isn't expected.

Senior (L5/E5): You should quickly establish the caching pattern (cache-aside vs read-through vs write-through) based on the access pattern and spend time on the real problems: cache invalidation strategy (TTL vs event-driven), thundering herd on cache miss (locking, request coalescing), cache key design and its impact on hit rate, and the staleness contract — how stale is acceptable and who signed off on it. You should quantify: "We cache product catalog with a 5-minute TTL because the business accepts 5 minutes of stale pricing in exchange for 10x lower DB load." Having an opinion on consistent hashing for cache distribution would be strong.

Staff+ (L6/E6+): You should dispatch the baseline architecture in 5 minutes and spend 25+ minutes on operational depth: multi-tier caching (L1 in-process → L2 Redis → L3 CDN), cache warming strategies for cold starts and deploys, the organizational question of who owns the staleness contract (engineering proposes TTLs, product signs off on user-facing staleness), and failure mode analysis — what happens when Redis goes down (do you fall through to DB and crush it, or serve stale from a local cache?). You should reason about cache sizing economics (memory cost vs DB query cost), hot key detection and mitigation (dedicated cache nodes or key splitting), and how caching intersects with consistency requirements across services. The interviewer should see you treat caching as a data freshness contract, not just a performance optimization.

1010. Staff Insiders: Controversial Opinions

These are uncomfortable truths that distinguish Staff engineers from Seniors. They're based on operating caches at scale, not on textbook knowledge. Strong engineers disagree on some of these — that's the point.

Most Stale Data Incidents Are Never Detected

The uncomfortable truth: Your cache is probably serving stale data right now. You just don't know it.

Why it's invisible:

The Staff position: If you can't measure staleness, you can't claim your cache is correct. Most teams measure cache performance but not cache correctness.

Bar-raiser question: "How would you know if your cache served incorrect data for the last hour?"

High Hit Rate Can Mask Correctness Bugs

The uncomfortable truth: 99% hit rate might mean you're serving confidently wrong answers 99% of the time.

Why this happens:

• Cache is fast, so nobody questions its answers
• Bugs in the origin path are never exercised
• Stale data looks like correct data if you don't check

Real-world example: A payment cache served 3-hour-old card data with 99.5% hit rate. Nobody noticed until a customer complained about being charged on a canceled card.

The Staff position: Hit rate is a performance metric, not a correctness metric. A cache with 99% hit rate that's 1% wrong can cause more damage than a cache with 80% hit rate that's always correct.

Removing a Cache Is Often the Right Fix

The uncomfortable truth: Many caching problems are best solved by removing the cache entirely.

Signs you should delete the cache:

• Hit rate below 50% (you're paying for misses)
• Write rate approaches read rate (invalidation churn)
• Consistency bugs that nobody can debug
• The origin can handle the load without the cache
• Multiple incidents traced to cache staleness

Why teams don't remove caches:

• "We already built it"
• "It must be helping somehow"
• Fear of origin load (often unfounded)
• Nobody owns the decision to remove

The Staff position: Adding a cache is easy. Removing one requires courage. The Staff engineer asks: "What if we just... didn't cache this?"

Cache Invalidation Is an Ownership Problem, Not a Technical One

The uncomfortable truth: "Cache invalidation is hard" is a cop-out. It's hard because nobody owns it.

Why invalidation fails:

The Staff position: Invalidation is hard because it's a coordination problem across teams, not a technical problem within one service. The fix is ownership clarity, not better algorithms.

Bar-raiser question: "Who is responsible for invalidation correctness across all services that write this data?"

Caches Amplify Bugs — They Don't Just Hide Them

The uncomfortable truth: A bug that affects 1% of requests without a cache might affect 99% of requests with a cache.

How caches amplify:

Without cache:
  - Bug writes bad data to DB
  - 1% of reads hit the bug
  - 99% read correct data from DB

With cache:
  - Bug writes bad data to DB
  - Bad data cached
  - 99% of reads hit cache → 99% see bad data

The Staff position: Caches turn transient bugs into persistent outages. A single bad write + aggressive caching = widespread incorrect data served for TTL duration.

"TTL Was Too Long" Is Never the Root Cause

The uncomfortable truth: When stale data causes an incident, "reduce TTL" is the wrong fix.

Why TTL tuning fails:

• It treats symptoms, not causes
• Shorter TTL = more origin load
• The real question: why wasn't invalidation triggered?

The Staff position: Every "TTL was too long" incident is actually an invalidation ownership failure. The fix is better invalidation, not shorter TTL. Shorter TTL is a band-aid that increases cost.

Real root causes:

• Writer didn't know to invalidate
• Invalidation code had a bug
• Invalidation was async and lost
• Nobody owned the cache contract

Access pattern: A B C D A B E F G H ...
LRU cache (size 4):
  After A B C D: [D, C, B, A]
  After A B:     [B, A, D, C]
  After E:       [E, B, A, D] (C evicted)
  After F:       [F, E, B, A] (D evicted)
  ...
  Hot keys A, B survive; scan keys E, F, G, H cycle through

Request arrives → Cache has stale entry
  → Return stale data immediately
  → Trigger background refresh
  → Next request gets fresh data

Cache miss:
  if (concurrent_origin_requests < limit):
    fetch from origin
  else:
    fail fast (or serve stale if available)

Write: UPDATE users SET email='new' WHERE id=123
Then:  DELETE cache:user:123

t=0: Thread A reads user:123 from DB (old data)
t=1: Thread B writes user:123 to DB (new data)
t=2: Thread B deletes cache:user:123
t=3: Thread A writes old data to cache
Result: Cache has stale data until TTL

Write: UPDATE users SET email='new' WHERE id=123
Then:  SET cache:user:123 = {new data}

Cache key: user:123:v7
On write: increment version → user:123:v8
Old cached data at v7 is naturally orphaned

Write: User updates profile
Set:   session.bypass_cache['user:123'] = now + 30s
Read:  If bypass active, read from DB, not cache

DB write → Changelog (binlog, WAL)
         → CDC consumer
         → Cache invalidation

cache_hit_total
cache_miss_total
cache_hit_rate (derived: hit / (hit + miss))
cache_latency_ms
origin_latency_ms
cache_eviction_total
cache_size_bytes
cache_key_count

Memory needed = working_set_size × (1 + overhead_factor)
Where:
  working_set_size = num_keys × avg_value_size
  overhead_factor ≈ 1.5-2x (for Redis data structures, fragmentation)

Cache size: 10%  → Hit rate: 70%
Cache size: 20%  → Hit rate: 85%
Cache size: 50%  → Hit rate: 95%
Cache size: 100% → Hit rate: 99%

These frameworks are referenced throughout this playbook and apply to many system design problems:

• →Distributed State Coordination

  • Cache invalidation coordination, multi-tier consistency, leader election for cache warming
  • Applies to: caching, rate limiting, locks, sessions

• →Degraded Mode Framework

  • Cache failure handling, serving stale vs failing, circuit breakers
  • Applies to: caching, rate limiting, dependency isolation

• →Build vs Buy Framework

  • Redis vs Memcached vs managed services, self-hosted vs cloud
  • Applies to: caching, observability, databases, queues