Module 7: Reliability & Failure Engineering Slides

Slide Outline

1. Reliability & Failure Engineering - Circuit breakers, bulkheads, graceful degradation, and chaos engineering — how reliability is engineered, not hoped for.
2. Learning Objectives - 5 outcomes for this module
3. Why This Module Matters - Reliability is what separates engineers who get woken up at 3am from engineers whose systems quietly do the right thing
4. Before vs After - The operational shift this module teaches
5. The Resilience Toolkit - Lesson section from the full module
6. Circuit Breakers in Detail - Lesson section from the full module
7. Bulkheads - Lesson section from the full module
8. Graceful Degradation - Lesson section from the full module
9. Chaos Engineering - Lesson section from the full module
10. Cascading Failures - Lesson section from the full module
11. Retry Amplification Visualised - Lesson section from the full module
12. Self-Check Quiz - Lesson section from the full module
13. Real-World Use Cases - Netflix's Hystrix (now retired) shaped the industry's circuit-breaker pattern; Resilience4j is the modern Java implementation., AWS uses bulkhead isolation extensively in their internal services to contain noisy-neighbour problems.
14. Common Mistakes to Avoid - 4 mistakes covered
15. Production Notes - 4 practical notes
16. Security Risks to Watch - 3 risks covered
17. Hands-On Labs - 3 hands-on labs
18. Key Takeaways - 5 points to remember

Learning Objectives

• Design retry policies that survive a downstream brownout
• Implement circuit breakers and understand the half-open state
• Apply bulkhead isolation to prevent noisy neighbours
• Build graceful degradation paths that turn outages into reduced functionality
• Run chaos experiments without breaking production

Why This Module Matters

Reliability is what separates engineers who get woken up at 3am from engineers whose systems quietly do the right thing during partial failure. The patterns in this module — circuit breakers, bulkheads, graceful degradation, chaos — turn outages into reduced functionality. The teams that adopt them ship faster (because they can deploy with confidence) and sleep better (because partial failure is contained, not amplified).

Production Notes

• Use error-rate-over-window for circuit-breaker tripping, not consecutive-failure count. Consecutive counts trigger on noise.
• Service meshes (Envoy/Istio/Linkerd) implement most resilience patterns at the data plane — do not rebuild them in application code if you have a mesh.
• Identify critical-path vs enriching dependencies; treat them differently in code (critical = propagate errors, enriching = swallow with logging).
• Run quarterly chaos game days. Untested resilience is hopeful resilience.

Common Mistakes

• Setting infinite retries on non-idempotent calls. One downstream blip becomes duplicate side effects everywhere.
• Cascading retries without budgets. A 3-hop chain with 3 retries each = 27x amplification on the failing service.
• No fallback for “enriching” calls (recommendations, sentiment, personalisation). Their failure should NOT fail the request.
• Chaos in production without practice in staging. The first real chaos experiment must not be your first chaos experiment.

Key Takeaways

• Reliability is engineered with timeouts, retries, circuit breakers, bulkheads, and graceful degradation — not hoped for
• Service meshes (Envoy / Istio) implement most of these patterns for free; use them
• Identify critical-path vs enriching dependencies and treat them differently in code
• Chaos engineering is the only way to know your resilience patterns actually work
• Cascading failures kill systems; break the chain with circuit breakers and retry budgets

Hands-On Labs

1. Lab 7.1 — Circuit Breaker in Action

   Implement a circuit breaker (Resilience4j or hand-rolled), trigger failure modes, validate state transitions.

   60 minutes - Intermediate

   • Wrap a flaky downstream call with a circuit breaker
   • Inject 50% error rate; observe breaker trip to OPEN
   • Wait for timeout; observe HALF-OPEN; success returns to CLOSED
   • Compare with no breaker: caller threads exhaust

   View lab files on GitHub

2. Lab 7.2 — Chaos Mesh on Kubernetes

   Run chaos experiments on a kind cluster and observe how the application reacts.

   120 minutes - Advanced

   • Install Chaos Mesh on a kind cluster
   • Inject pod-kill, network-loss, CPU stress experiments
   • Verify HPA, retry policies, circuit breakers all work
   • Document a chaos game-day runbook

   View lab files on GitHub

3. Lab 7.3 — Graceful Degradation Architecture

   Refactor a service with multiple dependencies to degrade gracefully under partial failure.

   60 minutes - Intermediate

   • Identify critical vs enriching dependencies
   • Add fallback responses for enriching dependencies
   • Inject failures and verify reduced-but-valid responses
   • Compare to baseline (full error on any dependency failure)

   View lab files on GitHub