Module 11: Real-World Failure Scenarios Slides

Slide Outline

1. Real-World Failure Scenarios - Retry storms, cache stampedes, split brain, hot partitions, queue overload, DNS outages, service-discovery failures, cascading failures — the incidents that actually happen, and how to engineer them away.
2. Learning Objectives - 5 outcomes for this module
3. Why This Module Matters - Real production engineers are recognised by the incidents they have absorbed and the runbooks they own. The patterns in
4. Before vs After - The operational shift this module teaches
5. Retry Storms - Lesson section from the full module
6. Cache Stampedes - Lesson section from the full module
7. Split Brain - Lesson section from the full module
8. Hot Partitions - Lesson section from the full module
9. Queue Overload - Lesson section from the full module
10. DNS Outages - Lesson section from the full module
11. Service Discovery Failures - Lesson section from the full module
12. Cascading Failures - Lesson section from the full module
13. Common Mistakes to Avoid - 4 mistakes covered
14. Production Notes - 3 practical notes
15. Security Risks to Watch - 3 risks covered
16. Hands-On Labs - 3 hands-on labs
17. Key Takeaways - 5 points to remember

Learning Objectives

• Recognise the canonical distributed-systems failure modes by their telemetry signatures
• Reproduce each failure in a controlled lab so the pattern is in your hands
• Apply the architectural defences that make each failure hard or impossible
• Write incident runbooks that an on-call engineer can actually use at 3am
• Run a post-incident review that produces lasting improvements

Why This Module Matters

Real production engineers are recognised by the incidents they have absorbed and the runbooks they own. The patterns in this module — retry storms, stampedes, split brain, hot partitions, queue overload, DNS outages, cascading failure — are the same outage taxonomy across every company at every scale. Engineers who internalise them respond in minutes; engineers who do not spend hours reconstructing what should have been recognised in the first thirty seconds.

Production Notes

• Build a per-failure-mode runbook library. Each runbook has detection signals, immediate-action checklist, recovery steps, and post-incident actions.
• Test runbooks in staging and chaos drills. Untested runbooks slow incident response, not speed it up.
• Capture every incident as a learning artefact even if there was “no real impact”. Near-misses are the cheapest training data.

Common Mistakes

• Skipping post-incident reviews on small incidents. The next bigger incident usually has the same root cause.
• Action items without owners or deadlines. The post-mortem becomes theatre.
• Treating retries as a fix instead of a load multiplier. Retries are a tool; budgets are the discipline.
• Single-cause root-cause analysis. Real incidents have multiple contributing factors; the post-mortem should surface all of them.

Key Takeaways

• Retry storms are caused by naive retries without budgets; cap them
• Cache stampedes need per-key locking, probabilistic expiration, or stale-while-revalidate
• Split brain is prevented by real consensus — do not invent HA without quorum math
• Cascading failures need defences at every layer: circuit breakers, budgets, load shedding, degradation
• Post-incident reviews convert incidents into engineering wins — without action items they are theatre

Hands-On Labs

1. Lab 11.1 — Reproduce a Retry Storm

   Configure naive retries; cause an outage; add backoff + budget; observe recovery.

   90 minutes - Intermediate

   • Set up 3-service chain with naive retries
   • Inject 50% errors on bottom service; observe storm
   • Add exponential backoff + budget; observe recovery

   View lab files on GitHub

2. Lab 11.2 — Cache Stampede on Expiry

   Cause a stampede when a hot key expires; add per-key locking; verify fix.

   60 minutes - Intermediate

   • Identify a hot key with TTL 30s
   • Send 1000 concurrent requests at expiry; observe origin meltdown
   • Add per-key Redis lock for recompute
   • Repeat; verify single recompute

   View lab files on GitHub

3. Lab 11.3 — Post-Incident Review

   Write a complete post-mortem for one of the simulated incidents above.

   60 minutes - Intermediate

   • Pick the retry-storm or cache-stampede incident
   • Reconstruct the timeline from logs/metrics
   • Document root cause, detection, mitigation
   • Define 3 concrete action items

   View lab files on GitHub