Kubernetes & Cloud Native Distributed Systems

Kubernetes is the substrate that runs most modern distributed systems. It is, itself, a distributed system — with consensus (etcd / Raft), partitioning (resources scheduled across nodes), replication (pods), and observability built in. Understanding how Kubernetes is constructed is now part of distributed-systems literacy.

Cluster Architecture

The control plane consists of:

• kube-apiserver: the only component that writes to etcd; every other component talks to apiserver. Stateless; horizontally scalable.
• etcd: the source of truth for cluster state; Raft-replicated; 3 or 5 nodes.
• kube-scheduler: assigns pending pods to nodes based on resource fit, affinity, and topology.
• kube-controller-manager: runs reconciliation loops (Deployment, ReplicaSet, Node, Endpoints, etc.). Each controller leader-elects via etcd.
• cloud-controller-manager: runs cloud-specific controllers (load balancers, persistent volumes, node lifecycle).

Each node runs:

• kubelet: the node agent; pulls container images, runs containers via the container runtime, reports node and pod status to apiserver.
• kube-proxy: implements Service abstraction via iptables / IPVS rules. (Modern alternative: Cilium with no kube-proxy.)
• CNI plugin: pod networking (Cilium, Calico, AWS VPC CNI, etc.). Provides pod IPs, NetworkPolicy enforcement, often eBPF observability.
• Container runtime: containerd or CRI-O; runs the containers.

Service, Ingress, Gateway API

• Service: stable virtual IP for a set of pods; load-balances internal traffic; ClusterIP for in-cluster, LoadBalancer for cloud LB, NodePort for external on a port.
• Ingress: L7 routing for HTTP/HTTPS; needs an Ingress Controller (nginx-ingress, AWS ALB, Envoy-based, etc.). Older API; many extensions baked into annotations.
• Gateway API: the modern replacement for Ingress; richer, role-separated (Gateway/HTTPRoute/etc.), portable across implementations. The right choice for new infra.

Service Mesh

A service mesh adds a sidecar proxy (Envoy) to every pod to handle service-to-service: mTLS, retries, circuit breaking, observability, traffic shifting, authorization. Three major options:

• Istio: most feature-rich; substantial complexity. Best when you need the full toolkit.
• Linkerd: simpler, performance-focused; written in Rust; zero-config mTLS. Best for “mesh basics, fast”.
• Cilium service mesh: eBPF-based, no sidecar, integrated with the Cilium CNI. Best when you want one tool for networking + mesh.

Service meshes implement most of the resilience patterns from Module 7 (timeouts, retries, circuit breakers) for free at the data plane. The cost is operational complexity and the latency tax of every request going through a sidecar.

Stateful Workloads

StatefulSets give pods stable identities (predictable name, predictable network address) and stable storage (PersistentVolumeClaims that follow the pod). The right pattern for databases, message queues, and any workload where pod identity matters.

StorageClasses define dynamic provisioning of PersistentVolumes from cloud-provider disks (EBS, PD, Azure Disk) or storage operators (Rook/Ceph, Longhorn). Choose access mode (ReadWriteOnce / ReadWriteMany), reclaim policy (Delete / Retain), and binding mode (Immediate / WaitForFirstConsumer) deliberately.

Autoscaling, PDBs, and Operational Sanity

• HPA: scale pods on metrics. Always set minReplicas >= 2 for HA.
• VPA: rightsize resource requests; clashes with HPA on the same metric.
• Cluster Autoscaler / Karpenter: scale nodes. Karpenter is the modern default on AWS.
• PodDisruptionBudget: cap the number of unavailable pods during voluntary disruption (drain, scale-down, eviction). Without PDBs, the autoscaler will happily evict every replica simultaneously.

Operational Practice

The Kubernetes operational discipline:

• Always run multi-AZ for production. Use topologySpreadConstraints to enforce it.
• etcd: 5 nodes across 3 AZs, KMS-backed encryption at rest, tested backup/restore.
• RBAC: deny by default; explicit allow per ServiceAccount; treat cluster-admin as root.
• NetworkPolicy: default-deny per namespace; explicit allow rules.
• PodSecurity admission: restricted profile by default, exceptions audited.

Module 8 of the Cloud Native Security Engineering course covers Kubernetes hardening in depth. The Kubernetes Security Simulator exercises the misconfigurations that cause real outages.

Service Mesh Traffic Flow

Self-Check Quiz

1. You have a Deployment with 3 replicas. The cluster autoscaler scales down a node. All 3 pods on that node get evicted. Why? (Answer: no PodDisruptionBudget. Define maxUnavailable: 1 so only one replica goes down at a time.)
2. Postgres in a Deployment vs StatefulSet — what changes? (Answer: StatefulSet gives stable pod identity (postgres-0, postgres-1) and stable PVCs that follow each pod. Required for any stateful workload.)
3. Service mesh adds 1ms latency per hop. Across 5 hops you pay 5ms. When is it worth it? (Answer: when the mesh-provided features (mTLS, retries, observability, traffic shifting) are worth more than 5ms. For mature production systems, almost always.)
4. You enable Istio mesh-wide STRICT mTLS on day one of rollout. What happens? (Answer: external load balancer health probes fail; non-meshed services can no longer talk to meshed services; outage. Phase: PERMISSIVE first, observe, promote namespace by namespace.)

For Kubernetes hardening, the Kubernetes Security cheatsheet is the operational reference. For service-mesh patterns the Service Mesh cheatsheet covers Istio/Linkerd/Cilium patterns — see the service mesh glossary entry for the conceptual definition. The Kubernetes cheatsheet is the day-to-day reference for kubectl operational patterns. Practice with the Kubernetes Security Simulator.