Design Service Discovery

How to Use This Playbook

This playbook supports three reading modes:

Layer                     Caches?     Respects TTL?
──────────────────────────────────────────────────
Application DNS cache     Sometimes   Often ignores (JVM: 30s default!)
OS resolver cache         Yes         Usually respects
Local DNS server          Yes         Respects
Upstream DNS server       Yes         Respects

GET /health    → 200 if process is alive (for restart decisions)
GET /ready     → 200 only if DB connected AND cache reachable AND <100 pending requests

What This Interview Actually Tests

Service discovery is not a "use Consul or Eureka" question.

This is a registry reliability and failure propagation question that tests:

• Whether you understand that the registry is a critical dependency for every service
• Whether you reason about what happens when the registry is wrong (stale entries, missing entries)
• Whether you design for registry failure (what if the registry itself is down?)
• Whether you understand the naming and routing abstractions that scale with organizational growth

The key insight: Service discovery failure doesn't cause one service to fail — it causes every service to fail. The registry is the most dangerous dependency in your microservices architecture because it's invisible until it breaks.

The L5 vs L6 Contrast (Memorize This)

Default Staff Positions (Unless Proven Otherwise)

The Three Intents (Pick One and Commit)

The Four Fault Lines (The Core of This Interview)

1. Client-Side vs Server-Side Discovery — Who resolves the address: the calling service or a load balancer?
2. DNS vs Registry vs Mesh — What's the lookup mechanism? Each has different consistency and latency tradeoffs.
3. Push vs Pull Health Checks — Does the registry actively check services, or do services self-report?
4. Centralized vs Distributed Registry — One registry cluster or per-region/per-domain registries?

Each fault line has a tradeoff matrix with explicit "who pays" analysis. See §3.

Quick Reference: What Interviewers Probe

Jump to Practice

→ Active Drills (§7) — 8 practice prompts with expected answer shapes

System Architecture Overview

Interview Walkthrough

The six phases below are compressed for a deep-dive format. Phases 1-3 deliver the crisp answer in 2-3 minutes. If probed, Phase 5 has depth for 15+ minutes.

Phase 1: Requirements & Framing (30 seconds)

Name the problem before the solution:

• "In a microservices architecture, services move — they scale up, scale down, get redeployed, fail, and recover. Hardcoded IP addresses don't work. Service discovery provides a dynamic registry so callers can always find healthy instances of any service."

Then frame the key decision:

• "The core tradeoff is client-side vs server-side discovery. Does the caller query a registry and pick an instance (client-side), or does a load balancer handle routing transparently (server-side)?"

Phase 2: Core Entities & API (30 seconds)

• Service Registry: the authoritative map of service name → healthy instances (IP:port)
• Service Instance: a running copy of a service with its address, health status, metadata (version, zone, weight)
• Health Check: active probe or heartbeat confirming an instance is alive and ready
• Watch/Subscription: a mechanism for callers to be notified of registry changes in real-time

Phase 3: The 2-Minute Architecture (2 minutes)

Phase 4: Transition to Depth (15 seconds)

"The basic approaches are well-understood. The hard problems are: registry consistency during network partitions, stale instance information causing requests to dead instances, and the interaction between service discovery and deployment (canary, blue-green)."

Phase 5: Deep Dives (5-15 minutes if probed)

Probe 1: "What happens during a network partition?" (3-5 min)

"The registry is a distributed system — typically backed by Raft consensus (Consul, etcd). During a network partition, the minority side of the partition can't elect a leader and becomes read-only."

Walk through the failure modes:

1. Caller in minority partition: The registry is read-only. New registrations and deregistrations don't propagate. The caller's cached instance list becomes stale. "Stale is better than empty — the caller should continue using its last known good list."
2. Service instance in minority partition: The instance can't renew its registration heartbeat. The registry marks it as unhealthy after the TTL expires. Other callers stop routing to it — even though the instance may be healthy within its partition.
3. Split-brain: Both sides of the partition think they're authoritative. After partition heals, the registry must merge — services that registered during the partition on both sides need reconciliation.

The key design decision: "AP vs CP for the registry. Consul and etcd are CP — they sacrifice availability during partitions (minority can't write). Eureka (Netflix) is AP — it sacrifices consistency (both sides accept registrations, may serve stale data). For service discovery, I prefer AP: serving slightly stale data is better than serving no data."

Probe 2: "How do you handle stale instances?" (3-5 min)

"A service instance crashes without deregistering. Its entry remains in the registry until the health check TTL expires (typically 30-90 seconds). During that window, callers route requests to a dead instance."

Mitigations:

1. Client-side health tracking: The caller tracks success/failure per instance. If 3 consecutive requests to instance X fail, remove it from the local cache — don't wait for the registry to catch up. "The caller detects failure in 3 requests (~1 second). The registry detects it in 30-90 seconds. Client-side detection is 30-90x faster."
2. Retry with next instance: On failure, immediately retry on a different instance. The caller never surfaces a single-instance failure to the user if healthy instances are available.
3. Fast deregistration: Use a shutdown hook to deregister on graceful shutdown. Only crashes leave stale entries. "In practice, 99% of instance removals are graceful deployments with shutdown hooks. Crashes are the 1% edge case."
4. Lease-based registration with short TTL: Register with a 10-second TTL and heartbeat every 5 seconds. A crashed instance disappears in 10 seconds instead of 90. "The cost: more heartbeat traffic. At 1,000 instances × 1 heartbeat/5s = 200 heartbeats/sec. That's trivial."

Probe 3: "How does service discovery interact with deployments?" (3-5 min)

"A canary deployment puts 5% of traffic on the new version. How does service discovery enable this?"

With client-side discovery:

• Register canary instances with metadata: version=2.1, canary=true
• Callers read metadata and route 5% of traffic to canary instances based on a hash or random selection
• "The routing logic is in the caller — which means every caller needs to support canary routing. Consistent behavior requires a shared client library."

With server-side discovery:

• The load balancer (Kubernetes Ingress, Envoy) handles weighted routing: 95% to stable, 5% to canary
• The caller doesn't know about the canary — routing is transparent
• "Simpler for callers. But the LB must support weighted routing and the deployment pipeline must configure it."

Phase 6: Wrap-Up

"Service discovery is the nervous system of a microservices architecture. The technology choice (Consul vs Kubernetes DNS vs Eureka) matters less than three operational decisions: (1) AP vs CP registry behavior during partitions, (2) client-side vs registry-side health detection speed, and (3) how discovery integrates with your deployment model. Get these three wrong and your services find each other in theory but fail to communicate in practice."

Quick-Reference: The 30-Second Cheat Sheet

1The Staff Lens

1.1 Why This Problem Exists in Staff Interviews

This is NOT a "pick a service registry" question. Everyone knows Consul exists.

This is a Registry Reliability & Failure Propagation question that tests:

• Whether you understand the registry as a single point of failure for the entire platform
• Whether you design for registry unavailability (graceful degradation, not hard failure)
• Whether you reason about stale discovery data (routing to dead instances, missing new instances)
• Whether you can design naming conventions that scale with organizational growth

1.2 The L5 vs L6 Contrast

Recall the five key behaviors from the Executive Summary. Below, we explain why each matters and what interviewers listen for.

Behavior 1: First move (ask about failure modes)

Behavior 2: Architecture (match mechanism to organizational needs)

Behavior 3: Failure reasoning (registry is a Tier-0 dependency)

Behavior 4: Health checking (multi-layer defense)

Behavior 5: Ownership (naming encodes organizational structure)

1.3 The Staff Question That Cuts Through Everything

This reframes the entire interview. The registry isn't the problem — the health authority is the problem. Staff engineers design for the case where the health authority is wrong, not just for the case where the registry is down.

2Problem Framing & Intent

2.1 The Three Intents

Before choosing any technology, ask What's the organizational complexity?

2.2 What's Intentionally Underspecified

The interviewer deliberately avoids specifying:

• Number of services and expected growth
• Container orchestration (Kubernetes vs bare metal vs hybrid)
• Multi-region requirements
• Security requirements (mTLS, zero-trust)
• Current pain points

Staff engineers surface these unknowns. Senior engineers jump to technology selection.

2.3 How to Open (The First 2 Minutes)

1. Ask about organizational scale and growth trajectory
2. State your mechanism assumption explicitly
3. Outline your plan: discovery mechanism → naming → health checking → failure modes → observability

Example opening:

3Fault Lines

3.1 Fault Line 1: Client-Side vs Server-Side Discovery

The tension: Client-side discovery gives services control over routing but requires client libraries. Server-side discovery is transparent but adds a hop.

L6 answer: "Server-side or sidecar, depending on platform maturity. For most teams, server-side (Kubernetes Services or a service-aware LB) is sufficient — services call a DNS name, the LB resolves to a healthy instance. For platforms needing fine-grained traffic control (canary, circuit breaking), the sidecar pattern (Envoy) gives control without client library complexity. I'd avoid client-side discovery unless we have a strong reason — maintaining client libraries in 5 languages is organizational debt."

L7 answer: "The pattern should match the organizational stage. Client-side is fine for 3 services in one language. Server-side scales to 50+ services without per-team effort. Sidecar/mesh is for 100+ services needing zero-trust and traffic management. Each migration adds capability and cost. I'd start with server-side and evolve to sidecar only when the pain justifies the operational investment."

3.2 Fault Line 2: DNS vs Registry vs Mesh

The tension: DNS is simple but has TTL staleness. Registries are dynamic but add infrastructure. Service mesh is comprehensive but expensive to operate.

L6 answer: "DNS-based with intelligent DNS (CoreDNS or Consul DNS interface) as the default. Services resolve checkout.payments.svc via DNS. Behind the scenes, DNS returns only healthy instances with short TTLs (5-10 seconds). This gives us health-aware routing with DNS simplicity. For services needing watch-based real-time updates, the registry API is available for direct consumption."

L7 answer: "I'd layer the mechanisms. DNS as the universal interface (every language/framework supports it). Registry API for services needing real-time updates (deployment orchestration, canary tooling). Service mesh for services needing mTLS and traffic shaping. Each layer is optional — teams opt in based on their requirements. This prevents forcing mesh complexity on teams that only need DNS."

3.3 Fault Line 3: Push vs Pull Health Checks

The tension: Active health checks (registry pings services) detect failures proactively but add load. Passive health checks (observe real traffic) have zero overhead but detect failures reactively.

L6 answer: "Hybrid: active health checks for process-level health (is the service reachable?), passive health checks for application-level health (is it returning correct responses?). The registry runs active TCP/HTTP checks every 5 seconds. The client-side LB or sidecar tracks real-time success rates — if a specific instance's error rate exceeds 50% over a 10-second window, remove it from the rotation immediately, don't wait for the next registry health check."

L7 answer: "I'd add a third layer: application-level readiness probes that reflect business health. A service can be TCP-healthy (process running), HTTP-healthy (returning 200), but business-unhealthy (database connection pool exhausted, returning errors for real requests). Readiness probes should check downstream dependencies, not just 'am I alive?' The three layers — liveness, readiness, and traffic-based — catch different failure classes."

3.4 Fault Line 4: Centralized vs Distributed Registry

The tension: A centralized registry is simple but is a single point of failure. Distributed registries are resilient but complex.

L6 answer: "Per-region registry clusters with cross-region federation. Each region has its own registry cluster — services in us-east register with the us-east registry. For cross-region discovery, registries federate: they share service catalogs asynchronously. If one region's registry fails, other regions continue independently. Cross-region lookups use the federated catalog with eventual consistency (acceptable — cross-region calls are already higher latency)."

L7 answer: "I'd separate the registry's control plane from its data plane. The control plane (consensus-based, for registration and deregistration) can be per-region. The data plane (serving lookups) should be distributed — cached locally on each node, refreshed from the control plane periodically. This way, the control plane can be briefly unavailable without affecting service-to-service communication."

4Failure Modes & Operational Reality

4.1 The Registry Cascade Failure

Timeline of a registry outage:

t=0:     Registry cluster loses quorum (AZ failure)
t=5s:    Services can't resolve new lookups
t=10s:   Client-side DNS cache starts expiring (TTL=10s)
t=15s:   Services with expired cache can't reach any other service
t=30s:   50 services report "service unavailable" errors
t=1m:    Every inter-service call fails
t=5m:    Registry quorum restored
t=5.5m:  Services re-resolve, traffic resumes

4.2 The Stale Registry Entry

4.3 The Naming Collision

The scenario: Two teams independently name their service user-service. Both register in the same registry. Traffic intended for Team A's service routes to Team B's service.

Staff position: This is an organizational problem, not a technical one. Enforce naming conventions with namespace requirements: {team}.{service}.{environment}. Registry registration validates namespace ownership — Team A can only register services under team-a.*. Naming collisions become impossible by construction.

4.4 Service Discovery in Deployments

During a rolling deployment:

Key concerns: (1) New instances must pass health checks before receiving traffic. (2) Old instances must drain connections before deregistering. (3) The registry must update quickly enough that clients don't route to terminated instances.

5Evaluation Rubric

5.1 Level-Based Signals

5.2 Strong Hire Signals

5.3 Lean No Hire Signals

5.4 Common False Positives

• Deep Consul/Eureka knowledge ≠ good discovery design (technology, not architecture)
• "Service mesh solves everything" ≠ understanding the operational cost
• Complex routing rules ≠ Staff thinking (often over-engineering)
• "Zero-downtime deployments" without explaining the discovery update mechanism

6Interview Flow & Pivots

6.1 Typical 45-Minute Structure

6.2 How Interviewers Pivot

6.3 What Silence Means

• After registry failure question → Interviewer wants specific degradation behavior, not just "we have HA"
• After naming question → Interviewer is testing organizational thinking, not technical naming
• After health check question → Interviewer wants you to distinguish process health from application health
• After deployment question → Interviewer wants to see awareness of the discovery-deployment interaction

6.4 Follow-Up Questions to Expect

1. "How do you handle a service that takes 30 seconds to start? When should it receive traffic?"
2. "Service A depends on Service B. Service B's registry entry is stale. How does Service A cope?"
3. "You have 200 services. How do you visualize and debug the dependency graph?"
4. "A team renames their service. How do you handle backward compatibility?"
5. "How do you support canary deployments through service discovery?"
6. "Your registry cluster runs out of memory. What happens?"

7Active Drills

Deep Dive 1: The Registry Outage

Context: Your Consul cluster (3 nodes) loses quorum at 2am. 80 services are unable to discover each other. Your on-call engineer has never dealt with a Consul failure before.

Questions You Should Ask First:

• Are services actually failing right now, or are they operating on cached discovery results — how much time do we have before caches expire?
• Why did the 3-node Consul cluster lose quorum — is this an AZ failure, disk issue, or network partition, and does the node placement tolerate single-AZ failure?
• Does the on-call engineer have a Consul failure runbook, or are they debugging from scratch at 2am?
• After recovery, should we upgrade to a 5-node cluster that tolerates 2 failures instead of 1?

Deep Dive 2: The Service Naming Migration

Context: Your company has 150 services with ad-hoc names (user-svc, my-api, service-v2-new-FINAL). Nobody knows who owns what. You need to migrate to a structured naming convention without downtime.

Questions You Should Ask First:

• Is this actually a technical DNS migration, or is it an organizational ownership discovery exercise disguised as a rename?
• Can we implement dual registration — both old and new names resolve simultaneously — so there's zero traffic disruption during migration?
• How do we track migration progress with data instead of deadlines — can we monitor old-name lookup counts and deprecate names only when they hit zero?
• What's the realistic timeline — is this 2 weeks of DNS changes, or 4-6 months of team coordination?

Deep Dive 3: The Thundering Herd on Discovery

Context: Your registry serves 500K lookups/sec. During a large-scale deployment (30 services deploying simultaneously), the registry sees a spike to 2M lookups/sec as clients refresh their caches. The registry becomes overloaded, causing lookup failures across the platform.

Questions You Should Ask First:

• Is the 4x traffic spike caused by synchronized cache invalidation across all callers during simultaneous deployments?
• Can we stagger deployments in waves of 5 with 2-minute gaps instead of deploying 30 services simultaneously?
• Are clients polling the registry on a fixed interval, or do they use random jitter so cache refreshes spread across a time window?
• Can we switch from polling to push-based mechanisms (Consul blocking queries, etcd watches) so clients don't need to poll at all?

Thundering herd on discovery: 30 simultaneous deployments trigger cache invalidation across all callers at once, creating a 4x traffic spike that overloads the registry. Fix: stagger deployments, add client-side refresh jitter, and switch to push-based updates.

Deep Dive 4: Multi-Region Service Discovery

Context: Your company operates in US and EU. Each region has its own service registry. A new requirement says that if all US instances of a service are unhealthy, traffic should automatically fail over to EU instances. How do you implement this?

Questions You Should Ask First:

• What's the failover priority chain — do we prefer local unhealthy instances with circuit breakers before falling back to cross-region, or go directly to the remote region?
• What's the latency increase when traffic fails over cross-region — have we communicated that users will see 5ms to 80ms latency increase as an explicit tradeoff?
• How does failback work when the local region recovers — do we shift traffic back gradually to prevent flapping, or instantly?
• Is the cross-region registry federation async with bounded staleness, or synchronous — and what's the acceptable sync interval?

Deep Dive 5: The Ghost Service

Context: A service was decommissioned 6 months ago but its registry entry was never cleaned up. Other services still have it in their dependency graphs. A new service is assigned the same port on the same host. Traffic intended for the decommissioned service now reaches the new service.

Questions You Should Ask First:

• Why did the registry entry persist for 6 months undetected — is there no TTL-based expiry or heartbeat requirement?
• Is service identity based on host:port (vulnerable to port reuse collisions) or unique instance identifiers (container ID, hostname+PID)?
• Do we have an automated registry audit that detects entries with no traffic — or does cleanup depend entirely on someone remembering a decommission checklist?
• Is service decommission defined as a multi-step process (stop traffic, verify zero connections, deregister, remove DNS, update dependency graphs), or a single event?

9Level Expectations Summary

After studying this playbook, you should be able to:

• Choose between DNS-based, registry-based, and mesh-based discovery based on organizational scale
• Design client-side caching for graceful degradation during registry outages
• Implement multi-layer health checking (liveness, readiness, traffic-based)
• Design naming conventions that encode team ownership and prevent collisions
• Walk through a registry failure scenario with timeline and blast radius analysis
• Explain the tradeoffs between client-side and server-side discovery patterns
• Design cross-region service discovery with health-aware failover

The Bar for This Question

Mid-level (L4/E4): Understands the fundamental problem service discovery solves — how services find each other when instances are ephemeral and IP addresses change. Can describe DNS-based or registry-based discovery at a high level and explain why hardcoded addresses don't work in dynamic environments. Knows that services register themselves and consumers look them up, even if the design lacks depth on failure modes.

Senior (L5/E5): Makes a deliberate choice between client-side discovery (consumer queries registry, picks an instance) and server-side discovery (load balancer queries registry, consumer hits a stable endpoint) with clear tradeoffs. Integrates health checks with registration so that unhealthy instances are removed from the registry promptly. Handles stale registrations from crashed services (TTL-based expiry, active health probes). Reasons about how rolling deployments interact with discovery — new instances registering before they're ready, old instances deregistering while draining connections.

Staff+ (L6/E6+): Identifies the service registry as the most invisible single point of failure in the platform — every service depends on it, but nobody thinks about its availability until it fails. Designs cache-based resilience so that services can continue routing to last-known-good endpoints during a registry outage. Addresses the bootstrapping problem: how does a service discover the registry itself? Thinks about organizational ownership — who runs the discovery platform, what SLOs govern it, and how is the cost of operating a consensus-backed registry justified to leadership.

10Staff Insiders: Controversial Opinions

10.1 "DNS Is Underrated for Service Discovery"

The conventional wisdom is that you need a dedicated service registry (Consul, Eureka) for microservices. The reality: DNS-based discovery covers 80% of use cases with zero additional infrastructure. Every language, framework, and tool supports DNS. Short TTLs (5-10 seconds) provide near-real-time updates. Health-aware DNS (CoreDNS with health checks, Consul DNS interface) provides the best of both worlds.

The experienced approach: start with DNS. Add a registry API only when you need features DNS can't provide (watches, KV store, ACLs). Most teams add a registry too early and pay the operational cost of running a consensus-based cluster for a problem that DNS solves.

And that's OK. Simple solutions that work are better than sophisticated solutions that require a dedicated team to operate.

10.2 "The Service Registry Is Your Most Invisible Single Point of Failure"

Everyone worries about database availability. Nobody worries about registry availability — until it fails and takes down every service simultaneously. The registry is the one dependency that every service shares, and it's invisible because it "just works" until it doesn't.

The uncomfortable truth: your microservices platform's availability is capped by your registry's availability. A 99.99% available payment service depending on a 99.9% available Consul cluster is effectively 99.9% available. Invest in registry resilience — 5-node clusters, cross-AZ deployment, client-side caching, automated recovery, and chaos testing.

Treat the registry like you treat the database: with respect, dedicated resources, and a runbook for 3am failures.

10.3 "Service Mesh Is the Right Answer for the Wrong Time"

Service mesh (Istio, Linkerd) provides everything: discovery, health checking, mTLS, traffic management, observability. It's the "right" answer architecturally. But the operational cost is enormous — sidecar proxies consume 10-15% of your compute, the control plane is a complex distributed system, and debugging mesh issues requires specialized expertise.

The counterintuitive truth: most companies adopt service mesh too early. They have 30 services, 3 teams, and no mTLS requirement — but they deploy Istio because it's the "modern" approach. The result: 6 months of mesh debugging instead of shipping features.

// GET /health (liveness)
{ "status": "ok" }

// GET /ready (readiness)
{
  "status": "ready",
  "checks": {
    "database": "connected",
    "cache": "connected",
    "downstream_auth": "reachable"
  }
}

// GET /health/detailed (debugging, not for load balancer)
{
  "status": "ready",
  "uptime_seconds": 86400,
  "version": "2.1.3",
  "connections": { "db": 45, "cache": 12 },
  "memory_mb": 256,
  "cpu_percent": 15
}

{domain}.{service-name}.{environment}[.{region}]

Examples:
  payments.checkout-api.prod.us-east-1
  catalog.search-service.staging
  identity.auth-gateway.prod
  platform.api-gateway.prod.eu-west-1