Design a Message Queue

A Staff+ playbook for system design interviews. This guide focuses on what separates L6/L7 answers from senior (L5) answers: delivery semantics, ordering tradeoffs, consumer design, and failure ownership — not just "use Kafka."

Executive Summary

How to Use This Playbook

1. Read the L5 vs L6 table to understand the calibration bar
2. Read the Interview Walkthrough to see how to present this in 45 minutes
3. Internalize the five Staff behaviors in Section 1.2
4. Practice the Active Drills in Section 7
5. Use the Appendices as reference during practice

System Architecture Overview

11. The Staff Lens

Why Message Queues Separate Staff from Senior

At first glance, message queues seem straightforward: producers write, consumers read, queues buffer. But in interviews, message queues are not about technology—they're about reliability contracts and organizational boundaries.

The Staff-level insight is this: Every queue is a promise about what happens when things go wrong. Do you lose data? Process it twice? Block producers? Shed load? The choice isn't right or wrong—it's about who absorbs the cost when the system is under stress.

Senior engineers gravitate toward Kafka or RabbitMQ and explain mechanics. Staff engineers ask: "What breaks if we lose a message? What breaks if we process it twice?" This question reveals:

• Whether you understand delivery semantics as business constraints, not technical features
• Whether you can reason about failure modes before they become production incidents
• Whether you think organizationally: who owns the dead letter queue? Who gets paged?

This isn't about Kafka vs SQS vs RabbitMQ. It's about whether you can design a reliability and ordering contract that survives production reality—and articulate the tradeoffs to make other engineers confident in your decisions.

1.1 The Bar: L5 vs L6 at a Glance

1.2 The Five Staff Behaviors

Behavior Comparison Table

Behavior 1: First move (clarify cost of failure)

Behavior 2: Ordering (avoid the parallelism trap)

Behavior 3: Delivery semantics (exactly-once is your job)

Behavior 4: Failure handling (poison messages need owners)

Behavior 5: Scaling (backpressure is the real question)

Default Staff Positions (Unless Proven Otherwise)

The Three Fault Lines

Every message queue interview resolves around three core tradeoffs. Naming them explicitly helps you structure your answers and anticipate interviewer probes:

These fault lines are explored in Sections 3-5. Each has a "who pays" tradeoff matrix.

Quick Reference: What Interviewers Probe

Jump to Practice

→ Active Drills (§7) — 8 practice scenarios with expected answer shapes

Interview Walkthrough: How to Present This in 45 Minutes

This section bridges the gap between HelloInterview-style step-by-step guides and our Staff-level analysis. Senior candidates spend 25 minutes explaining what Kafka is and run out of time before reaching anything interesting. Staff candidates speed through the baseline in 10-12 minutes — fast enough to spend the remaining 30+ minutes on the delivery semantics, partition strategy, and operational ownership questions that actually determine your level.

The six phases below add up to 45 minutes. The ratios matter: phases 1-4 are deliberately compressed so phase 5 gets the lion's share of time. If you're spending more than 12 minutes before the transition to depth, you're pacing like an L5.

Phase 1: Requirements & Framing (2-3 minutes)

State functional requirements in 30 seconds — don't enumerate, state the category:

• "We need asynchronous, durable message delivery between producer and consumer services with ordering guarantees and replay capability."

That's it. Don't list every message type or consumer.

Invest time on non-functional requirements (this is the Staff move):

• "What's the delivery guarantee requirement? At-least-once for most systems, exactly-once only when downstream is non-idempotent. I'll design for at-least-once because exactly-once pushes complexity to the wrong layer."
• Clarify: ordering scope (per-entity vs global?), throughput target (10K vs 1M msgs/sec?), retention (hours vs days vs indefinite replay?)
• "I'll assume at-least-once with per-entity ordering, because that covers 90% of production message queue use cases and lets me make clean partition key decisions."

Phase 2: Core Entities & API (1-2 minutes)

State entities quickly (30 seconds):

• Topic — the named channel for a class of events (e.g., order-events, payment-results)
• Partition — the unit of parallelism and ordering; events with the same key land in the same partition
• Producer — publishes messages with a partition key that determines routing
• ConsumerGroup — logical subscriber; each partition assigned to exactly one consumer in the group
• Message — key + payload + headers + timestamp + offset
• Offset — the consumer's position in the log; the mechanism that enables replay and the place where exactly-once breaks down

API (1 minute) — publish/subscribe, not request/response:

produce(topic, key, message) → ack
consume(topic, group_id) → messages[]
commit_offset(topic, group_id, partition, offset) → ack

The key parameter on produce is the most important design decision in the entire API. It determines partition assignment → ordering scope → parallelism ceiling. Everything flows from the key.

Admin operations (not latency-sensitive):

create_topic(name, partitions, replication_factor, retention)
reset_offsets(topic, group_id, target)  // replay after consumer bugs

Phase 3: High-Level Architecture (5-7 minutes)

Draw the producer → broker → consumer flow with partition assignment visible:

┌──────────┐                                ┌──────────────┐
│ Producer  │──── key: order_123 ──────────▶│  Partition 0  │──▶ Consumer A
│ Service   │──── key: order_456 ──────────▶│  Partition 1  │──▶ Consumer B  ── Consumer
└──────────┘──── key: order_789 ──────────▶│  Partition 2  │──▶ Consumer C     Group
                                            └──────┬───────┘
                                                    │ retry exhausted
                                            ┌───────▼──────┐     ┌──────────┐
                                            │  Dead Letter  │────▶│  On-Call  │
                                            │    Queue      │     │  Alert   │
                                            └──────────────┘     └──────────┘

Walk the interviewer through the request flow (reference the full System Architecture diagram above for the complete picture):

"Producer services publish events to Kafka with a partition key. Kafka hashes the key to route to a specific partition — 12 partitions with replication factor 3 for durability. Each partition is assigned to exactly one consumer in the group, giving us parallelism across partitions while preserving per-entity ordering. Consumers commit offsets after successful processing. If a message fails after 3 retries, it moves to the DLQ with an alert."

Key points to hit on the whiteboard:

1. Kafka for durability — replicated, append-only log; messages survive broker failures
2. Partitioning for parallelism — partition count is the parallelism ceiling; consumer count ≤ partition count
3. Consumer groups for scaling — each partition gets exactly one consumer; add consumers up to partition count
4. Schema registry — Avro/Protobuf schema enforcement prevents producer bugs from poisoning consumers
5. DLQ for operational safety — messages that fail after retries go to DLQ, not back to the main queue

Then immediately flag the key tension: "This works for ordered, at-least-once delivery within a partition. The interesting questions are: what happens when a consumer dies mid-batch and the partition rebalances? Who owns the poison pill policy when a malformed message blocks the queue? And how do you handle the exactly-once illusion when your consumer crashes between processing and committing?"

Phase 4: Transition to Depth (1 minute)

At this point you have a correct, simple architecture on the board. Now you pivot:

"The basic architecture is well-understood — Kafka gives us durable, partitioned, ordered message delivery. What makes this Staff-level is the reliability reasoning. Let me dive into three areas: (1) the exactly-once illusion and why it's a consumer-side problem, (2) partition rebalancing storms and how they cause duplicates, (3) DLQ governance as an organizational problem."

Then offer the interviewer a choice:

"I can go deep on any of these. Which is most interesting to you?"

If the interviewer doesn't have a preference, lead with the exactly-once illusion — it's the most universally asked and the most misunderstood.

Phase 5: Deep Dives (25-30 minutes)

The interviewer will steer, but be prepared to go deep on any of these. For each, follow the Staff pattern: state the tradeoff → pick a position → quantify the cost → explain who absorbs that cost.

Fault Line 1: The exactly-once illusion (5-7 min)

Open with the insight:

"Exactly-once delivery doesn't exist. What exists is at-least-once delivery with idempotent consumers — which gives you effectively-once processing. The queue can't guarantee exactly-once because there's a gap between 'consumer processes message' and 'consumer commits offset.' If the consumer crashes in that gap, the message replays."

Go deeper — walk through the idempotency mechanism:

1. Consumer receives message with order_id: 123 and event_id: abc-456
2. Before processing, check Redis: SETNX idempotency:{event_id} 1 EX 86400
3. If key exists → skip (already processed). If key doesn't exist → process → commit offset
4. The idempotency window (24h TTL) must exceed the maximum replay window

The Staff follow-up: "Kafka Transactions (exactly-once semantics in Kafka) solve a narrow problem: read-process-write within Kafka. The moment your consumer writes to PostgreSQL or calls an external API, you're back to at-least-once + idempotency. Don't confuse Kafka's internal EOS with end-to-end exactly-once."

Cross-reference §3.1-3.3 Delivery Semantics for the full analysis.

Fault Line 2: Consumer rebalancing storms (5-7 min)

"When a consumer dies or a new one joins, Kafka triggers a rebalance — all partitions in the group get reassigned. During rebalance, no consumer can commit offsets. If the rebalance takes 30 seconds and you have a high-throughput topic, you get: (a) 30 seconds of buffered messages that replay after rebalance, (b) duplicate processing for any messages consumed but not committed before the rebalance started, (c) possible ordering violations if a partition moves to a consumer with different in-flight state."

Name the mitigations:

• Cooperative rebalancing (KIP-429): only revoke affected partitions, not all partitions
• Sticky assignment: prefer reassigning partitions to the same consumer after rebalance
• Static group membership: assign a group.instance.id so planned restarts don't trigger rebalance
• Commit frequency: commit every N messages or every T seconds — lower T means fewer duplicates on rebalance but more commit overhead

Quantify: "With eager rebalancing on a 50-partition topic processing 10K msgs/sec, a single consumer restart causes ~30s of reprocessing across all partitions — that's 300K duplicate messages. With cooperative rebalancing + static membership, the same restart only reprocesses the 10 partitions that consumer owned — maybe 60K duplicates."

Fault Line 3: Partition key design and hot partitions (3-5 min)

"If you key by user_id and one user generates 50% of events (a batch job, a test account, a viral user), one partition gets 50% of the load while the others are idle. Your parallelism ceiling drops from N partitions to effectively 1."

The Staff answer: "You have three options: (a) composite key (user_id + event_type) to spread load at the cost of per-user ordering, (b) monitoring for partition skew with alerts, (c) the nuclear option — repartition the topic with more partitions, which requires a carefully orchestrated migration."

Fault Line 4: DLQ governance — the organizational problem (3-5 min)

"After 3 retries with exponential backoff, the message moves to the DLQ. But a DLQ without an owner is a black hole."

Walk through the governance model:

1. Ownership: The consuming team owns the DLQ for their consumer group — they investigate and resolve
2. SLA: Investigate DLQ messages within 4 hours, replay or discard within 24 hours
3. Classification: Is it a transient failure (retry will succeed), a data bug (producer sent bad data), or a code bug (consumer can't handle valid data)?
4. Replay tooling: reset_offsets to replay from DLQ to main topic after fixing the consumer bug
5. Alerting: DLQ ingestion rate > 0 for > 10 minutes → PagerDuty alert to the consuming team

The Staff insight: "90% of DLQ messages are code bugs, not transient failures. If your DLQ is growing, your consumer has a bug. Don't keep retrying — fix the consumer, then replay."

Operational maturity: monitoring and alerting (3-5 min)

"The three metrics that matter for a message queue: (1) consumer lag — the gap between the latest produced offset and the latest committed offset per partition; (2) consumer group throughput — messages processed per second per consumer; (3) DLQ ingestion rate — if this is non-zero, something is broken."

Name the alert thresholds: "Consumer lag > 10K messages for > 5 minutes → page the owning team. DLQ ingestion rate > 0 for > 10 minutes → P2 incident. Consumer group throughput drops > 50% → investigate rebalance or consumer crash."

The Staff insight: "The most dangerous failure mode is a consumer that's technically running but processing slowly. Lag grows linearly, no alerts fire because the consumer is 'healthy,' and by the time someone notices, you have hours of backlog. That's why you alert on lag, not just consumer health."

Phase 6: Wrap-Up (2-3 minutes)

Summarize the key tradeoff — don't just restate your architecture, synthesize the insight:

"Message queues are reliability contracts, not technology choices. The Staff-level challenge is: who absorbs the cost of imperfection? For at-least-once delivery, consumers absorb the cost through idempotent handlers. For ordering, the partition key selection determines the scope — and you trade parallelism for ordering guarantees. For failure handling, the DLQ governance model determines whether failures get investigated or silently accumulate."

If time permits, add the organizational insight:

"The harder problem is ownership. When a message fails, whose pager goes off — the producer team who sent a malformed message, or the consumer team whose handler can't process it? In my experience, you need both: the schema registry catches producer bugs at publish time, and the DLQ owner investigates consumer failures. Without clear ownership, the DLQ becomes a graveyard."

Common Timing Mistakes

Reading the Interviewer

What to Deliberately Skip

These topics are traps. L5 candidates spend time on them. Staff candidates name them, dismiss them, and redirect to what matters.

→ Continue to Fault Lines (§3) for the Staff-grade tradeoff reasoning.

With the Staff lens established—queues as reliability contracts, not technology choices—we now move to execution. The next sections break down the three fault lines that define every queue design.

22. Problem Framing & Intent

Before drawing boxes, name the intent. Message queues serve three distinct purposes with different requirements:

L6 (Staff) answer: Names intent before architecture. "Are we building async task processing (delivery matters), event sourcing (ordering + durability), or load leveling (backpressure)? Each has different tradeoffs."

L7 (Principal) answer: Identifies when you need multiple systems: "Event sourcing for the source of truth, separate task queues for side effects, with clear boundaries between them."

When NOT to Use a Queue (Staff Candidates Say No)

Staff candidates win interviews by knowing when to not use a queue. This is high-signal behavior.

Do NOT use a queue when:

Bar-Raiser Follow-up: "When would you tell the team NOT to use a queue here?"

Expected answer: "If the caller needs a synchronous response, if the volume is low enough that direct calls work, or if the debugging/tracing cost of async exceeds the decoupling benefit — I'd push back on adding queue complexity."

33. The Fault Lines

Fault Line 1: Lose messages vs process duplicates. With intent clarified, we address the first core fault line: what happens when the network fails, consumers crash, or brokers restart? Delivery semantics answer "did it arrive?" and determine who absorbs the cost of imperfection.

3.1 The Three Guarantees

3.2 The Exactly-Once Myth

Why exactly-once is hard:

1. Producer crashes after send, before ack → duplicate on retry
2. Consumer crashes after process, before commit → duplicate on restart
3. Network partition → both sides think they're right

Staff solution: At-least-once delivery + idempotent consumers.

3.3 Idempotency Patterns

L6 (Staff) answer: "For payment processing, we'll use at-least-once delivery with a deduplication table keyed by idempotency token. The consumer checks the table before processing and writes the result + token atomically."

Fault Line 2: Ordering scope determines parallelism ceiling. How strictly do events need to arrive in order? The answer constrains consumer parallelism and determines who pays when hot keys appear.

3.4 Ordering Spectrum

3.5 Choosing Partition Keys

Hot partition problem: If one entity generates disproportionate traffic (celebrity user, large enterprise tenant), that partition becomes a bottleneck.

Staff solutions:

• Accept the imbalance if it's rare
• Sub-partition: user_id + sequence_number % N
• Route hot keys to dedicated infrastructure

44. Failure Modes & Degradation

Delivery and ordering establish the contract. Now we design for when that contract is violated: component failures, poison messages, and the critical question of who gets paged when things break.

4.1Failure Catalog

Message:     {offset, partition_key, timestamp, payload}
Partition:   append-only log of messages
Offset:      {consumer_group, partition} → last_committed_offset

Dedup table: {idempotency_key} → {processed_at, result}
TTL:         retention window (e.g., 7 days)

4.2Dead Letter Queue Design

DLQ requirements:

• Preserve original message + metadata (timestamp, retry count, error)
• Searchable by error type, time range, partition key
• Replay capability (back to main queue or specific consumer)
• Alerting on growth rate, not just size

L6 (Staff) answer: "DLQ is not a black hole. We need: (1) alerting when DLQ grows faster than 10/min, (2) a dashboard showing failure reasons, (3) a replay tool that can target specific time ranges, (4) clear ownership — the team that owns the consumer owns its DLQ."

4.3The Ownership Question

Staff consideration: If you can't name the owner, the failure mode is unhandled. In interviews, explicitly state ownership for each failure path you describe.

4.4Backpressure & Flow Control

Fault Line 3: The backpressure decision — where Staff and Senior diverge most. Ownership answers "who handles failures." Backpressure answers "what happens when producers outpace consumers"—the question that separates candidates who've operated queues in production from those who've only designed them on whiteboards.

This is the message queue equivalent of Rate Limiter's "fail-open vs fail-closed" decision. Senior engineers focus on steady-state throughput. Staff engineers design for the moment when the queue backs up — and it always backs up eventually.

The Backpressure Question

This is the question many Senior (L5) candidates miss entirely. When producers outpace consumers, you must choose who fails: producers (throttled), consumers (overwhelmed), or end users (degraded experience). There is no neutral choice.

Bounded Queues

Staff consideration: Unbounded queues are a lie. Memory is finite. Choose your failure mode explicitly: block, reject, or drop.

Consumer Lag Monitoring

Key metrics:

• Consumer lag: Messages waiting to be processed
• Lag growth rate: Is lag increasing or stable?
• Processing time: Time from enqueue to process complete

Alert thresholds:

• Lag > X minutes → Warning (scale consumers)
• Lag growth rate positive for > Y minutes → Critical (investigate)
• Processing time > SLA → Page on-call

The Degraded Mode Decision

L6 (Staff) answer: "For notifications, we'll drop oldest when the queue exceeds 100K — a 30-minute-old 'your order shipped' notification is worthless. Product has signed off that we'd rather lose stale notifications than block order processing. We'll alert when drop rate exceeds 1% so we can scale proactively."

55. Evaluation Rubric

5.1Level-Based Signals

5.2Strong Hire Signals

5.3Lean No Hire Signals

5.4Common False Positives

• Knows Kafka internals deeply: Deep Kafka knowledge ≠ good queue design. Candidates who focus on partitions/offsets but miss delivery semantics are Senior, not Staff.
• Draws complex event flows: Complexity isn't a Staff signal. Simple partition ordering + idempotent consumers beats complex exactly-once coordination.
• Mentions many queue technologies: Breadth without tradeoffs (Kafka vs SQS vs RabbitMQ) is encyclopedic, not Staff-level.

66. Interview Flow & Pivots

6.1Typical 45-Minute Structure

6.2How Interviewers Pivot

6.3What Silence Means

• After delivery semantics question: Interviewer wants you to reason about lost vs duplicate messages
• After "what else?": You're missing failure modes, ownership, or backpressure
• After definitive answer: They may want you to consider the opposite (e.g., "always use queues" → when NOT to use queues)

6.4Follow-Up Questions to Expect

1. "How do you ensure idempotent message processing?"
2. "What happens when a consumer processes a message but crashes before committing the offset?"
3. "How do you handle a message that fails processing 100 times?"
4. "What if one partition gets 90% of the traffic?"
5. "How do you test dead letter queue handling in production?"
6. "What metrics would you alert on?"
7. "How do you replay messages from the DLQ?"

6.5Queue-Specific Traps

Trap 1: Claiming exactly-once without idempotency

• Red flag: "Kafka's exactly-once setting handles it"
• Staff correction: "Exactly-once requires idempotent consumers. The queue gives at-least-once; we build idempotency."

Trap 2: Ignoring partition hot keys

• Red flag: "We'll partition by user_id"
• Staff correction: "If one user generates 90% of events, that partition becomes a bottleneck. We need hot key detection or consistent hashing."

Trap 3: Unbounded retry

• Red flag: "We'll retry until it succeeds"
• Staff correction: "Poison messages retry forever. We need a retry budget (3-5 attempts) and a DLQ with ownership."

77. Active Drills

Practice these scenarios to internalize Staff-level thinking. Try answering before revealing the Staff approach.

These scenarios test Staff-level operational thinking. Unlike drills (which test interview responses), deep dives test ownership reasoning — the kind of thinking that happens when you're the Staff engineer responsible for the system.

Deep Dive 1: Consumer Lag Incident

Deep Dive 2: Poison Message Flooding DLQ

Deep Dive 3: Producer Timeout During Peak

Deep Dive 4: Ordering Violation Incident

Deep Dive 5: Consumer Crash Loop

99. Level Expectations Summary

What gets you each level in a message queue interview:

What Separates Each Level

Quick Self-Check

Before your interview, verify you can answer:

• What are the three delivery semantics, and when would you choose each?
• Why does ordering only apply within a partition, and how do you design for it?
• What's your DLQ strategy, and who owns investigation?
• When the queue backs up, what do you drop — and who signs off?
• How do you achieve exactly-once semantics without exactly-once delivery?

The Bar for This Question

Mid-level (L4/E4): You should be able to choose between a message broker (RabbitMQ, SQS) and an event log (Kafka) with basic reasoning, design a producer-consumer architecture with acknowledgment, and explain why messages can be lost without acks. You can describe at-least-once vs at-most-once delivery. Understanding partition-level ordering or dead letter queues would be a bonus but isn't expected.

Senior (L5/E5): You should quickly establish the delivery contract (at-least-once with idempotent consumers) and spend time on the hard problems: partition key design for ordering guarantees, consumer group rebalancing during deploys, DLQ strategy with investigation ownership, and backpressure when consumers fall behind. You should be able to explain why exactly-once delivery is impossible but exactly-once processing is achievable through idempotency. Having a clear opinion on Kafka vs SQS for the specific use case — with tradeoffs, not just preference — would be strong.

Staff+ (L6/E6+): You should dispatch the architecture in 5 minutes and spend the remaining time on organizational and operational depth: schema evolution strategy (Avro with a registry vs freeform JSON and its downstream cost), cross-team event contracts (who owns the schema, who is responsible when a consumer breaks on a schema change), poison message handling across team boundaries, and the build-vs-buy decision for event infrastructure. You should reason about the operational cost of running Kafka (dedicated team, broker upgrades, partition rebalancing) vs the limitations of a managed service (SQS ordering constraints, SNS fan-out limits). The interviewer should walk away understanding that message queues are an organizational coordination problem, not just a technology choice.

1010. Staff Insiders: Controversial Opinions

These are uncomfortable truths that distinguish Staff engineers from Seniors. They're based on operating queues at scale, not on textbook knowledge. Strong engineers disagree on some of these — that's the point.

"Exactly-Once" Is Mostly a Lie

The uncomfortable truth: When someone says "exactly-once delivery," they almost always mean something weaker.

Why it's a lie:

Where it actually breaks:

• Multi-producer: Two services both think they're the source of truth. Both publish. Consumer gets duplicates.
• Rebalance: Consumer processes message, dies before committing offset. New consumer reprocesses.
• Replay: You intentionally reprocess messages. "Exactly-once" becomes "exactly-twice."
• Cross-system: You can't have exactly-once between Kafka and Postgres. Period.

The Staff position: Stop saying "exactly-once" unless you can explain the precise boundary. Most systems should be designed for "effectively-once": at-least-once delivery + idempotent consumers that make reprocessing safe.

The bar-raiser question: "Walk me through what happens when your consumer processes a message, writes to the database, and crashes before committing the offset. How do you prevent the duplicate write?"

Backpressure Beats Buffering (When Queues Become Liabilities)

The uncomfortable truth: A growing queue is not "absorbing load." It's hiding a problem that will explode later.

The failure cascade:

t=0:    Producers: 10K/s, Consumers: 8K/s → Queue grows
t=1hr:  Queue: 7.2M messages, lag: 12 minutes
t=2hr:  Queue: 14.4M messages, memory pressure, broker slows
t=3hr:  Broker GC pauses, producers timeout, data loss
t=4hr:  Incident declared. "The queue failed."

The lie: "The queue gives us time to scale." The truth: the queue gave you time to not notice you were sinking.

When buffering helps:

• Short bursts: Traffic spike < buffer size, consumers catch up during lull
• Bounded queues: Explicit limit, producers back off when full

When buffering hurts:

• Sustained overload: Queue grows forever, you're just delaying the crash
• Unbounded queues: "Let the queue absorb it" with no limit = time bomb

The Staff position: Backpressure (reject early, let producers handle it) is often better than buffering (accept everything, hope consumers catch up). The question is: who can afford to wait?

• If producers can wait: bounded queue + producer backoff
• If producers can't wait: drop or degrade early, don't pretend you're handling it
• If nobody can wait: you have a capacity problem, not a queue problem

The bar-raiser question: "Your queue is at 10M messages and growing. What's your triage process? At what point do you declare an incident vs 'wait for autoscaling'?"

"Let Teams Own Their Consumers" Often Fails

The uncomfortable truth: Distributed consumer ownership sounds empowering. In practice, it often leads to operational chaos.

What goes wrong:

The pager dilution problem: When 10 teams each own a consumer, who gets paged for:

• Broker outage?
• Topic compaction misconfigured?
• Schema incompatibility?
• Partition rebalance storm?

Answer: Either everyone (alert fatigue) or no one (silent failure).

The Staff position: Consumer ownership works when:

1. Clear boundaries: Teams own consumers, platform team owns broker + topics + schemas
2. Standards: Shared consumer library with built-in metrics, DLQ, circuit breakers
3. Central observability: One dashboard, one alerting pipeline, clear escalation
4. Schema governance: Breaking change process, not "YOLO publish"

When to centralize: If you're seeing operational fragmentation (different alerting, different DLQ policies, no cross-team visibility), the "decentralized" model has failed. Consider a platform team that owns the messaging infrastructure.

The bar-raiser question: "Team A's consumer is lagging and blocking Team B's messages on a shared topic. Who owns the incident? What's the escalation path?"

Related Frameworks

These cross-cutting frameworks apply to message queue design and appear in other playbooks:

• →Coordination Strategy Framework

  • Local-first vs centralized, consistency bounds
  • Applies to: consumer offset management, partition assignment

• →Degraded Mode Framework

  • What happens when the queue is down or slow?
  • Applies to: producer behavior during outages, consumer lag handling

• →Build vs Buy Framework

  • Self-hosted Kafka vs managed Confluent vs SQS
  • TCO analysis, operational burden assessment