Design a Load Balancer

How to Use This Playbook

If you have 2 hours before your interview, read the Interview Walkthrough and §3 (Fault Lines). Everything else is depth you can pull on where you're weak. Appendices are collapsed — expand them for targeted review.

Executive Summary

What This Interview Actually Tests

Load balancing is not a "just add nginx" question. Everyone knows round-robin.

This is a distributed systems ownership question that tests:

• Whether you understand the L4 vs L7 tradeoff and when each matters
• Whether you reason about health checking failure modes proactively
• Whether you recognize session affinity as a scalability anti-pattern
• Whether you can design for load balancer failure itself

The key insight: Load balancing is a single point of failure disguised as a reliability feature. Staff engineers reason about what happens when the "reliable" component fails.

The L5 vs L6 Contrast (Memorize This)

Default Staff Positions

These are your opening stances. Adjust based on requirements.

System Architecture Overview

Interview Walkthrough

The six phases below are compressed for a deep-dive format. Phases 1-3 fit in 2-3 minutes. If the interviewer keeps probing, you expand into Phase 5's detailed fault lines. Most interviews don't go beyond the first probe — know when to stop talking.

Phase 1: Requirements & Framing (30 seconds)

Name the intent before drawing a single box:

• "Load balancing serves three purposes: distribute traffic for horizontal scaling, detect and route around unhealthy instances, and enable zero-downtime deployments."

Then immediately frame the four decisions:

• "Four things matter: L4 vs L7, algorithm selection, health checking strategy, and LB redundancy. Let me walk through each."

Phase 2: Core Entities & API (30 seconds)

State the components (not entities — this is infrastructure):

• VIP (Virtual IP): the stable endpoint clients connect to; maps to the load balancer
• Backend pool: set of healthy server instances, each with weight and health status
• Health check: active probe (HTTP GET /health) + passive monitoring (error rate tracking)
• Connection drain: graceful removal of a backend — finish in-flight requests before cutting traffic

Phase 3: The 2-Minute Architecture (2 minutes)

Deliver this in ~90 seconds. Hit the four key decisions and move on:

Then stop. Let the interviewer steer.

Phase 4: Transition to Depth (15 seconds)

If the interviewer wants more, offer choices:

"I can go deeper on any of these four. The most interesting tradeoffs are: health check tuning and cascading failure, sticky sessions as a scalability trap, or what happens to in-flight requests when the LB itself fails."

Phase 5: Deep Dives (5-15 minutes if probed)

Probe 1: "What happens when a health check is wrong?" (3-5 min)

Walk through the fix:

1. Blast radius protection: Never remove more than 20% of the fleet at once. If >20% are failing health checks, assume the problem is the health check, not the servers. "Panic mode: if half your fleet fails health checks simultaneously, the health check is lying."
2. Gradual drain, not instant removal: When a server fails health checks, reduce its weight over 30 seconds before removing. This gives time for false positives to self-correct.
3. Health check circuit breaker: If the health check endpoint itself is slow (because the server is under load), don't fail the check — that creates a death spiral. Separate the "is the server alive?" check from the "is the server healthy?" check.

Probe 2: "What about sticky sessions?" (3-5 min)

Walk through the alternatives:

1. Best answer: Externalize session state to Redis. Any server handles any request. Sticky sessions become unnecessary.
2. If sticky sessions are unavoidable (legacy app, WebSocket connections): Use consistent hashing so adding/removing servers only remaps ~1/N sessions. "With 10 servers, adding one remaps ~10% of sessions. Round-robin remaps 100%."
3. WebSocket sticky sessions: These are legitimate — the connection IS the session. Use a connection registry (Redis hash: connection_id → server_id) so other services know where to route messages for a specific connection.

Probe 3: "What if the load balancer itself fails?" (3-5 min)

1. Active-passive pair: Two LB instances, one active. VRRP failover in 1-5 seconds. During failover, active TCP connections are reset — clients must reconnect.
2. Active-active with ECMP: Multiple LB instances sharing the same VIP via BGP/ECMP. The network distributes packets across all instances. No failover — just capacity reduction when one fails.
3. Cloud-managed LB: AWS ALB/NLB, GCP Cloud LB. Already multi-AZ redundant. "The LB is someone else's problem — but you still need to reason about cross-region failure."

The deeper question: "What happens to in-flight requests? L4 LBs reset TCP connections on failover — the client sees a connection timeout and must retry. L7 LBs can retry transparently for idempotent requests. Clients need retry logic regardless."

Probe 4: "How do you handle zero-downtime deployments?" (3-5 min)

"Connection draining is the key. When removing a server for deployment: (1) stop sending NEW requests, (2) let existing requests complete (drain timeout = p99 request duration × 2), (3) once all connections are closed or timeout expires, shut down the server."

"The reverse — bringing a new server online — uses slow-start. Don't immediately give it full traffic weight. Ramp from 10% to 100% over 30 seconds. This lets the new server warm up (JIT compilation, cache population) before taking full load."

Phase 6: Wrap-Up

If you've gone deep, close with the organizational insight:

"Load balancing is the most over-engineered and under-monitored part of most architectures. Teams spend weeks choosing between nginx and HAProxy, then never set up alerting for connection drain failures or health check false positives. The Staff question isn't 'which load balancer' — it's 'what are the failure modes, who gets paged, and what does the runbook say.'"

Quick-Reference: The 30-Second Cheat Sheet

Key Numbers Worth Memorizing