Module 5: Consensus & Coordination Slides

Slide Outline

1. Consensus & Coordination - How distributed nodes agree — Raft, Paxos, leader election, distributed locking, and the etcd / ZooKeeper / Consul systems that production runs on.
2. Learning Objectives - 5 outcomes for this module
3. Why This Module Matters - Engineers who understand consensus stop being scared of etcd. They can read a Raft log replay, recover a stuck cluster,
4. Before vs After - The operational shift this module teaches
5. What Consensus Is - Lesson section from the full module
6. Raft - Consensus for Humans - Lesson section from the full module
7. Paxos and Why You Probably Don't Implement It - Lesson section from the full module
8. Cluster Sizing - Lesson section from the full module
9. Distributed Locking - Lesson section from the full module
10. etcd, ZooKeeper, Consul - The Production Trio - Lesson section from the full module
11. Operational Hazards - Lesson section from the full module
12. Distributed Lock with Fencing Token - Lesson section from the full module
13. Real-World Use Cases - etcd backs every Kubernetes cluster ever deployed; Raft is in your production stack whether you knew it or not., CockroachDB runs thousands of Raft groups per cluster, one per data range.
14. Common Mistakes to Avoid - 3 mistakes covered
15. Production Notes - 3 practical notes
16. Security Risks to Watch - 4 risks covered
17. Hands-On Labs - 3 hands-on labs
18. Key Takeaways - 5 points to remember

Learning Objectives

• Explain consensus as a problem and why it is fundamental to CP systems
• Walk through Raft leader election, log replication, and safety in detail
• Compare Raft and Paxos and pick between them in practice
• Implement distributed locking correctly (with fencing tokens, not just SETNX)
• Operate etcd, ZooKeeper, or Consul without taking down your cluster

Why This Module Matters

Engineers who understand consensus stop being scared of etcd. They can read a Raft log replay, recover a stuck cluster, and design a system around the trade-offs of CP versus AP rather than tripping over them. This is the module that separates engineers who treat distributed coordination as “magic” from engineers who treat it as load-bearing infrastructure they own.

Production Notes

• Always use odd-number Raft clusters: 3 for most workloads, 5 for high availability. Never even.
• Practice etcd snapshot recovery quarterly. The runbook for recovering a stuck quorum is the difference between minutes and hours of cluster downtime.
• Run etcd on dedicated SSDs with low fsync latency. Slow disks slow every write across the cluster.

Common Mistakes

• Inventing your own “HA” with naive locks (Redis SETNX) instead of using consensus primitives. Always fails under partition.
• Adding a node to a stuck Raft cluster instead of restoring from snapshot. New nodes need a quorum to join.
• Running consensus on shared infrastructure (etcd on the same disk as your database). Slow neighbour = stuck quorum.

Key Takeaways

• Consensus is the substrate of every CP system; Kubernetes, Vault, and most modern infra rely on Raft etcd
• Use Raft for new systems; Paxos is the older, harder alternative
• Cluster size: 2N+1 tolerates N failures; always odd; 3 or 5 for nearly all real workloads
• Distributed locking requires consensus + fencing tokens, not naive SETNX
• A stuck quorum is recoverable but only if you have a tested runbook; never improvise

Hands-On Labs

1. Lab 5.1 — etcd Cluster Bootstrap and Failover

   Run a 3-node etcd cluster, write data, kill one node, verify availability; kill two, observe stuck quorum.

   90 minutes - Intermediate

   • Bootstrap 3-node etcd via docker-compose
   • Write keys, observe replication
   • Kill one node; verify writes still succeed
   • Kill two nodes; verify writes block
   • Restore from snapshot; re-add members

   View lab files on GitHub

2. Lab 5.2 — Distributed Lock with Fencing Token

   Implement a correct distributed lock using etcd Lease + fencing token; demonstrate why naive locks fail.

   90 minutes - Advanced

   • Implement naive Redis SETNX lock; reproduce partition failure mode
   • Implement etcd-based lock with fencing token
   • Resource (Postgres) rejects operations from old tokens
   • Demonstrate two-client race; only one operation succeeds

   View lab files on GitHub

3. Lab 5.3 — Leader Election in Application Code

   Build a singleton-task pattern: many replicas, only one runs the periodic job at a time.

   60 minutes - Intermediate

   • Use Kubernetes Lease object as election primitive
   • Run 3 replicas; only the leader executes the cron logic
   • Kill the leader; verify another replica takes over within seconds

   View lab files on GitHub