Design a Distributed Message Queue

How to Use This Playbook

Organized for interview use first, reference second. Read front-to-back once. Return to individual sections for targeted review.

The L5 vs L6 Contrast — Start Here

What This Interview Actually Tests

Message queues are not a "use Kafka" question. This is a reliability contract and organizational ownership question that tests:

• Whether you understand delivery semantics as business constraints, not technical features
• Whether you reason about failure modes before they become production incidents
• Whether you think organizationally: who owns the DLQ? Who gets paged? What's the SLA?
• Whether you can articulate when NOT to use a queue

The Staff Positions

The Three Intents

Name the intent before designing anything.

What Interviewers Probe

Quick-Reference: The 30-Second Cheat Sheet

System Architecture Overview

Phase 1: Frame the Problem (1 minute)

Before drawing anything, surface the two questions that change the entire architecture.

Say this:

• "Two questions before I start: What's the cost of losing a message? What's the cost of processing a message twice? For notifications, a duplicate push is annoying — for payment charges, a duplicate is catastrophic. These two answers determine delivery semantics, idempotency requirements, and DLQ governance."

Commit to intent:

• "I'll assume an order processing pipeline: at-least-once delivery because lost orders are unacceptable, ordering within the same order_id, and idempotent consumers because duplicates during broker recovery would cause duplicate charges."

Phase 2: When Not to Use a Queue (30 seconds)

Name this explicitly — it's a high-signal behavior.

Say this:

• "Before I add queue infrastructure: Is the caller waiting for a response? If yes, a queue adds latency with no benefit. Is the volume low enough for direct service calls? Every queue is an operational burden, a consistency downgrade, and a debugging tax. For this use case, async processing is clearly the right call because the caller doesn't wait and we need load leveling. But I always want to name this check."

Phase 3: The 3-Minute Architecture (3 minutes)

Walk four layers: producers, topic design, consumer groups, downstream idempotency.

1. Producers. Producers publish with acks=all (wait for all in-sync replicas) and an idempotent producer configuration (dedup at the broker level). Key by business entity — order_id for order events — so all events for the same order land on the same partition in the same order. Schema validated against the Schema Registry before publish; breaking changes rejected at publish time, not discovered later by consumers.

2. Topic design. 12 partitions gives us 12 parallel consumer instances as the ceiling. Replication factor 3 for broker fault tolerance. Separate DLQ topic (orders.dlq, 3 partitions) with the same RF. Retention policy is business-driven: 7 days for replay capability, longer for compliance-driven topics.

3. Consumer groups. Multiple consumer groups read the same topic independently — fulfillment and analytics are fully decoupled. Each group manages its own committed offset. Fulfillment: 3 workers, each owning 4 partitions. Analytics: 2 workers with at-most-once semantics — stale click events are worthless; we accept the loss. Each group scales independently up to the partition count ceiling.

4. Idempotency. Fulfillment workers check Redis for idem:{order_id}:{event_type} before processing. If the key exists: the message was already processed — return the cached result, commit the offset, move on. If it doesn't: process, write to PostgreSQL and Redis atomically (or as close as possible), commit offset. This makes duplicate delivery from broker recovery or rebalancing safe.

Phase 4: Transition to Depth (15 seconds)

Signal where the hard problems are.

"The happy-path architecture is straightforward. The hard problems are: the exactly-once myth and idempotency design, the ordering tax and hot partition risk, DLQ governance, and the backpressure decision when consumers fall behind. Which do you want to go into?"

Phase 5: Deep Dives When Probed (5–15 minutes)

Probe A: "What happens when a consumer crashes mid-processing?"

"This is the core of at-least-once delivery. Consumer reads message, begins processing, writes to the database, crashes before committing the offset. On restart, Kafka redelivers the message at the last committed offset. The consumer processes it again. The database write happens again. Without idempotency, that's a duplicate.

With idempotency: before any write, check the dedup table keyed by message_id or idempotency_key. If it exists: we already processed this — return the cached result and commit. If it doesn't: write to the database and the dedup table in the same transaction (or with the dedup write first). Commit the offset last. Now a crash after the transaction but before the offset commit just triggers a dedup-cache hit on restart. Safe."

Probe B: "Why not global ordering? Seems simpler."

"Global ordering means a single consumer — or a single partition acting as a global sequencer. That's your parallelism ceiling: 1. For a system processing 10K orders/second, a single consumer is a throughput wall and a single point of failure.

What actually needs ordering? Events for the same order. Order #123 must see: Created → Payment → Fulfilled — in that order. Order #456 can process completely independently. Partition by order_id: all events for order #123 land on the same partition, processed by the same consumer, in guaranteed order. Events for different orders process in parallel across all partitions. You get ordering where it matters plus parallelism everywhere else."

Probe C: "What about hot partitions from one entity generating disproportionate traffic?"

"If one user generates 50% of events, one partition gets 50% of load. Three options: accept the imbalance if it's rare (monitor and address if it becomes chronic), sub-partition with a composite key like user_id + event_type % N to spread load at the cost of strict per-user ordering, or route identified hot keys to dedicated over-provisioned partitions. The right answer depends on whether the hot key is a transient anomaly or a structural characteristic of the data."

Probe D: "Consumer lag is growing. 2M messages backed up. What do you do?"

"Consumer lag is a symptom, not the disease. Before I do anything: is lag growing (sustained overload requiring a decision) or stable (burst that will self-resolve)? These have completely different mitigations.

Growing lag: why can't we keep up? Check consumer error rate, processing time p99, and external dependency latency — the most commonly missed root cause. If processing is slow because a downstream database is slow, adding consumers doesn't help. If consumers are healthy and the issue is genuine throughput: scale horizontally up to the partition count, then decide on the backpressure policy. Who signs off on dropping oldest messages if we can't catch up within SLA? That's not an engineering decision — it's a product decision that needs cross-functional alignment before the incident, not during it."

Phase 6: Close with Constraints (30 seconds)

"Message queues are reliability contracts, not technology choices. The Staff challenge is: who absorbs the cost of imperfection? At-least-once delivery means consumers absorb the cost through idempotent handlers. Partition ordering means you trade global correctness for parallelism. DLQ governance means the consuming team absorbs investigation cost. The backpressure decision means someone — product, engineering, or users — absorbs the cost of overload. Name every one of these before the incident, not during it."

1The Staff Lens

1.1 Why Message Queues Separate L5 from L6

At first glance, message queues seem straightforward: producers write, consumers read, queues buffer. In interviews, message queues are not about technology — they are about reliability contracts and organizational boundaries.

Senior engineers gravitate toward Kafka or RabbitMQ and explain mechanics. Staff engineers ask: "What breaks if we lose a message? What breaks if we process it twice?" This question reveals whether you understand delivery semantics as business constraints, whether you reason about failure modes before production incidents, and whether you think organizationally about who owns each failure path.

1.2 The L5 vs L6 Contrast — Visual

1.3 The Key Insight

2Problem Framing & Intent

2.1 The Three Intents

Intent 1: Async Processing (Email sending, image resize, notification dispatch)

• Delivery matters; ordering usually doesn't
• At-least-once is the default; idempotent consumers handle duplicates
• Backpressure strategy: shed oldest stale events during overload

Intent 2: Event Sourcing (Audit logs, CQRS, compliance)

• Ordering within entity and durability are critical
• Replay is a first-class requirement — log retention period is a business decision
• Immutable append-only log; compaction only for snapshot use cases

Intent 3: Load Leveling (Traffic spikes, batch pipelines, click stream)

• Throughput and backpressure handling dominate
• At-most-once may be acceptable if events are individually low-value
• Consumer lag SLA must be defined before incidents, not during them

2.2 When NOT to Use a Queue

Staff candidates win points by knowing when to say no. A queue is wrong when:

2.3 What the Interviewer Leaves Underspecified

Interviewers deliberately omit:

• Delivery semantics (at-most-once, at-least-once, exactly-once behavior)
• Ordering scope (none, partition, global)
• Durability requirements (can we lose messages?)
• Throughput shape (steady load vs burst)
• Who owns the DLQ

Staff engineers surface these. Senior engineers assume them away.

3Fault Lines

3.1 Fault Line 1: Delivery Semantics

The tension: At-most-once is simple and fast but loses messages. Exactly-once is impossible end-to-end without consumer-side idempotency. At-least-once with idempotent consumers is the practical default for most production systems.

L6 answer: At-least-once delivery with idempotent consumers. Dedup table keyed by message_id or idempotency token, checked before processing. Write result and dedup key atomically, then commit offset. Crash-safe: on restart, the dedup check catches the duplicate before any side effect occurs.

3.2 Fault Line 2: Ordering Guarantees

The tension: Global ordering gives perfect consistency but requires a single consumer — a parallelism ceiling and SPOF. Partition ordering scales horizontally and is almost always sufficient because users only perceive ordering within a single entity's events.

L6 answer: Partition ordering by entity ID. All events for Order #123 partition to the same key → same partition → same consumer → guaranteed order. Orders A, B, and C process in parallel across partitions. You get ordering where users perceive it (within an order) plus parallelism everywhere else (across orders).

3.3 Fault Line 3: Backpressure Strategy

The tension: When producers outpace consumers, you must choose who fails: producers (throttled, adding latency), consumers (overwhelmed, lagging), or end users (degraded experience). Unbounded buffering defers the choice until it becomes a crisis. There is no neutral option — only explicit and implicit choices.

L6 answer: Define the backpressure policy before the incident, with cross-functional sign-off. For notifications: drop oldest when queue exceeds 100K — a 30-minute-old "your order shipped" notification is worthless. Product has signed off. Alert when drop rate exceeds 1% so we can scale proactively before the drop policy kicks in. For order processing: block producers before losing any events. For analytics: sample and surface "approximate data during overload" to users.

4Failure Modes & Operational Reality

4.1 Failure Catalog

4.2 Dead Letter Queue — Design and Governance

DLQ requirements:

• Preserve original message + metadata: timestamp, retry count, error type, original partition/offset
• Searchable by error type, time range, and partition key
• Replay capability: back to main topic (after consumer fix) or to a test consumer (to validate the fix)
• Alert on growth rate, not absolute size — 100/min for payments is a P1 incident; 10/day is background noise

L6 answer: "DLQ is not a black hole — it's a failure evidence store. Requirements: (1) Alert on DLQ growth rate, not absolute size. (2) Dashboard showing failure reason distribution — 90% of DLQ messages are code bugs, not transient failures. (3) Replay tooling that can target specific time ranges or error types. (4) Named owner: the consuming team owns DLQ investigation; the platform team owns DLQ tooling and replay infrastructure."

4.3 The Ownership Matrix

4.4 Consumer Rebalancing — The Hidden Duplicate Source

When a consumer dies or a new one joins, Kafka triggers a rebalance — all partitions in the group get reassigned. During rebalance, no consumer can commit offsets.

The cost: With eager rebalancing on a 50-partition topic at 10K msgs/sec, a single consumer restart causes ~30s of reprocessing across all partitions — 300K potential duplicate messages.

Mitigations:

• Cooperative rebalancing (KIP-429): Only revoke affected partitions, not all partitions. 300K duplicates → ~60K.
• Static group membership: Assign group.instance.id so planned restarts don't trigger rebalance at all.
• Commit frequency: Commit every N messages or every T seconds — lower T means fewer duplicates on rebalance but more commit overhead.
• Idempotent consumers: The actual fix — makes all of the above safe rather than just reducing the blast radius.

5Evaluation Rubric

5.1 Level-Based Signals

5.2 Strong Hire Signals

5.3 Lean No-Hire Signals

5.4 Common False Positives

• Deep Kafka internals knowledge ≠ queue design. Candidates who focus on log segments and controller elections but miss delivery semantics are Senior, not Staff.
• Complex event flow diagrams ≠ Staff signal. Simple partition ordering + idempotent consumers beats complex exactly-once coordination.
• Breadth of queue technologies ≠ Staff thinking. Kafka vs SQS vs RabbitMQ without tradeoffs is encyclopedic, not analytical.

6Interview Flow & Pivots

6.1 Typical 45-Minute Shape

6.2 How Interviewers Pivot — And What They're Testing

6.3 Follow-Up Questions to Expect

1. "How do you ensure idempotent message processing?"
2. "What happens when a consumer processes a message but crashes before committing the offset?"
3. "How do you handle a message that fails processing 100 times?"
4. "What if one partition gets 90% of the traffic?"
5. "How do you replay messages from the DLQ after fixing the consumer bug?"
6. "What metrics would you alert on and at what thresholds?"
7. "Team A's consumer is slow and creating lag. How does this affect Team B on the same topic?"

7Active Drills

Deep Dive 1: Consumer Lag Incident

Context: Order processing queue has 2M messages backed up. Lag is growing at 10K/min. On-call escalates to you.

Questions to Surface First:

• Is lag growing (sustained overload requiring a backpressure decision) or stable (burst that will self-resolve)? The mitigation is completely different.
• What's the SLA for order processing latency? Are we in breach, or do we have a window?
• Is one consumer group lagging or all of them? Is this a specific partition hot-spot?
• What changed recently — deployment, schema change, upstream data format change, upstream traffic pattern?

Deep Dive 2: DLQ Flooding for Payment Events

Context: DLQ for payment processing is growing at 100 messages/minute. What do you do?

Questions to Surface First:

• When did DLQ growth start? Does it correlate with a deployment, data migration, or upstream schema change?
• Is this one root cause or multiple independent failures? Sample the error type distribution.
• What's the business impact per minute of unprocessed payments?
• Who owns DLQ investigation — the infrastructure team or the payments team?

Deep Dive 3: Producer Timeout During Flash Sale

Context: During a flash sale, your event producer starts timing out. Analytics events are being dropped. The analytics team is complaining.

Questions to Surface First:

• Which data is being dropped — transactional events (orders) or observability events (analytics clickstream)?
• Is the flash sale a predictable event that should have been capacity-planned?
• Who absorbs the cost of overload: the producer (blocks, adds user-facing latency), the broker (buffers, risks cascade), or the data (drops)?

Deep Dive 4: Ordering Violation Incident

Context: A customer complains their order was "shipped" before it was "paid." Events arrived out of order despite being sent in order.

Questions to Surface First:

• Was the order shipped without verifying payment status (consumer state machine bug) or did the shipping service trust event ordering (architecture bug)?
• Producer-side bug (events sent out of order) or consumer-side bug (events processed out of order)?
• What's the partition key strategy? Are all events for the same order going to the same partition?
• How many other event flows might have this same class of bug?

Deep Dive 5: Consumer Crash Loop

Context: A consumer keeps crashing. It restarts, processes a few messages, crashes again. The same messages keep getting redelivered.

Questions to Surface First:

• Is it always the same message causing the crash (poison message) or different messages (systemic bug)?
• What changed recently — new deployment, schema change, upstream data format change?
• What's the consumer's error handling strategy? Does it have a per-message retry limit?

Without a circuit breaker: consumer crashes → restarts at same offset → same poison message → crash → infinite loop. With per-message failure counter: 3 retries → DLQ → alert → investigation. The process never crashes.

9Level Expectations Summary

After studying this playbook, you should be able to:

• Name the two questions that determine delivery semantics before drawing any architecture
• Explain why exactly-once delivery is a consumer-side responsibility, not a queue feature
• Design idempotent consumers with a concrete dedup mechanism
• Articulate why global ordering kills parallelism and design partition ordering with hot key mitigations
• Define a DLQ governance model with named ownership, investigation SLA, and replay tooling
• Explain the backpressure options (drop/block/degrade) and name who signs off on each
• Know when NOT to use a queue and be able to name the three costs

The Bar for This Question

Mid-level (L4/E4): Choose between a message broker and event log with basic reasoning. Design a producer-consumer architecture with acknowledgment. Explain why messages can be lost without acks. Understand at-least-once vs at-most-once.

Senior (L5/E5): Quickly establish the delivery contract (at-least-once with idempotent consumers) and spend time on hard problems: partition key design for ordering guarantees, consumer group rebalancing during deploys, DLQ strategy with investigation ownership, and backpressure when consumers fall behind. Clear opinion on Kafka vs SQS with tradeoffs.

Staff+ (L6/E6+): Dispatch the architecture in 5 minutes and spend the remaining time on organizational depth: schema evolution strategy (Avro with registry vs freeform JSON and its downstream cost), cross-team event contracts (who owns the schema, who is responsible when a consumer breaks on schema change), poison message handling across team boundaries, and build-vs-buy for event infrastructure. You should know when NOT to use a queue and be able to push back on the premise. The interviewer should walk away understanding that message queues are an organizational coordination problem, not just a technology choice.

10Staff Insiders: Controversial Opinions

10.1 "Exactly-Once" Is Mostly a Lie

When someone says "exactly-once delivery," they almost always mean something weaker.

Where it actually breaks: Multi-producer (two services both publishing); rebalance (consumer processes, dies, new consumer reprocesses); replay (intentional reprocessing makes "exactly-once" mean "exactly-twice"); cross-system (you cannot have exactly-once between Kafka and PostgreSQL — they don't share a transaction log).

The Staff position: Stop saying "exactly-once" unless you can name the precise boundary. Design for "effectively-once": at-least-once delivery + idempotent consumers that make reprocessing safe. This is achievable and honest.

10.2 Backpressure Beats Buffering

A growing queue is not "absorbing load" — it is hiding a problem that will explode later.

t=0:    Producers: 10K/s, Consumers: 8K/s → Queue grows at 2K/s
t=1hr:  Queue: 7.2M messages, lag: 12 minutes
t=2hr:  Queue: 14.4M messages, memory pressure, broker slows
t=3hr:  Broker GC pauses, producers timeout, data loss begins
t=4hr:  Incident declared. "The queue failed."

The lie: "The queue gives us time to scale." The truth: The queue gave you time to not notice you were sinking.

The Staff position: Backpressure (reject early, let producers handle it) is often better than buffering (accept everything, hope consumers catch up). The question is: who can afford to wait? Name your answer explicitly before the incident.

10.3 "Let Teams Own Their Consumers" Often Fails

Distributed consumer ownership sounds empowering. In practice it often leads to operational chaos.

The pager dilution problem: When 10 teams each own a consumer, who gets paged for a broker outage? Topic compaction misconfigured? Schema incompatibility? Either everyone (alert fatigue) or no one (silent failure).

The Staff position: Consumer ownership works when: platform team owns broker + topics + schemas; service teams own consumers + DLQs; a shared consumer library provides built-in metrics, DLQ, and circuit breakers; and central observability gives one dashboard with clear escalation paths. If you're seeing operational fragmentation (15 different alerting setups), decentralization has already failed.