Real-World Failure Scenarios

Every distributed system fails the same way the others do. The taxonomy is small and the failure modes are well-documented, which means the patterns that defend against them are equally well-understood. The engineers who experience these incidents and build the defences are the ones the rest of the org calls when something is on fire.

This module walks the canonical incidents one at a time: what they look like, what causes them, how to defend.

Retry Storms

Symptom: a downstream service has a brownout. Errors trigger client retries. Retries multiply load on the failing service. The service cannot recover. p99 latency stays high; error rate stays high; you cannot get out of it without a deploy or restart.

Defence: every retry policy has a budget (cap retries at, say, 10% of total RPS). Exponential backoff with jitter. Circuit breakers stop calling the dead service so it can recover.

Cache Stampedes

Symptom: a hot cache key expires under load. Hundreds of concurrent requests miss the cache and hit the origin. Origin overloads. Stays overloaded until the cache is repopulated.

Defence: per-key locking on cache misses (only one recompute at a time). Probabilistic early expiration. Stale-while-revalidate semantics. The Caching Strategies guide covers all three patterns.

Split Brain

Symptom: a network partition isolates two halves of a CP cluster. Both elect leaders independently. Both accept writes. When the partition heals, you have divergent state.

Defence: real consensus algorithms (Raft, Paxos) require a majority quorum, so only one side can elect a leader; the other side cannot make progress. The lesson: do not invent your own “HA” without consensus underneath.

Hot Partitions

Symptom: one shard / partition / Redis slot receives 10x the traffic of the others. That node saturates while others sit idle. p99 latency on the hot key climbs; the rest of the system looks fine.

Defence: detect via per-partition QPS metrics. Mitigate by salting keys, splitting the hot key into N keys, or fronting with a per-pod local cache so the hot key never reaches the distributed cache.

Queue Overload

Symptom: producers outpace consumers. Queue depth grows. Eventually the queue runs out of memory or disk; messages start failing or get dropped.

Defence: backpressure. Bounded queues. Producer throttling on consumer-lag signals. Auto-scale consumers on lag.

DNS Outages

Symptom: DNS resolver is slow or unreachable. Every service-to-service call stalls on lookup. The cluster appears to be hanging without errors.

Defence: NodeLocal DNSCache to keep DNS off the critical path. Short TTLs combined with negative-caching tuning. Service-mesh-based discovery (sidecar handles endpoint changes via xDS, no DNS in the data path).

Service Discovery Failures

Symptom: discovery system (Consul, etcd, kube-apiserver) is unhealthy. Services cannot find each other. Existing connections work; new connections fail.

Defence: clients cache the last-known-good endpoint set with a generous TTL. The system tolerates a degraded discovery system if existing connections can survive the window.

Cascading Failures

The mother of all distributed-systems incidents. A small failure becomes a cluster-wide outage because every layer amplifies the load. The diagram above shows the basic shape: DB slows, Service B times out, Service A retries, the gateway queues up requests, users retry, the gateway saturates, more services fail.

Defences: circuit breakers per dependency. Retry budgets capping amplification. Load shedding (return 503 to a percentage of requests when saturated). Graceful degradation paths so a non-critical failure doesn't block critical paths. The Incident Response Simulator walks through real scenarios.

The Post-Incident Review

The blameless post-mortem is the discipline that turns incidents into learning. The structure that works:

1. Timeline: minute-by-minute account of what happened. No interpretation; just facts.
2. Impact: who was affected, for how long, in what way.
3. Root cause: what enabled the incident. Often multiple contributing factors.
4. Detection: how was the incident discovered? Could it have been earlier?
5. Mitigation: what stopped the incident? Was it the right action?
6. Action items: each one assigned, deadlined, tracked. Without these the document is theatre.

Cache Stampede Visualised

Split Brain Architecture

Self-Check Quiz

1. A retry storm is happening. You add exponential backoff. Symptoms partially improve. What did you miss? (Answer: backoff alone helps but does not cap total RPS to the failing service. Need a retry budget too.)
2. Your cache stampedes every 5 minutes for 2 seconds. The TTL is 5 minutes. What is the simplest fix? (Answer: probabilistic early expiration — a small chance to recompute before TTL — distributes load over time without coordination.)
3. You operate two regions in active-active. Both elect leaders during a partition. Why is this catastrophic for payments? (Answer: split brain. Both sides accept conflicting writes. Without consensus, reconciliation requires manual resolution. Active-active works for read-heavy data, not writes-with-consequence.)
4. Post-mortem action items consistently slip. What changes the dynamic? (Answer: assigned owner, deadline, tracked alongside feature work, reviewed in next post-mortem. Without follow-up the document is theatre.)

The Runtime Security cheatsheet covers detection patterns for the failure scenarios above. Practice with the Incident Response Simulator.