Module 9: Secrets Management & Machine Identity

Learning Objectives

• Integrate HashiCorp Vault with Kubernetes
• Implement dynamic secrets and automatic rotation
• Replace static credentials with workload identity
• Design a secrets management strategy for production

Why This Matters

Secret sprawl is consistently in the top 3 causes of cloud-native breaches. Every leaked API key, every long-lived certificate, every hardcoded password is a potential headline. This module teaches you to systematically eliminate static secrets and replace them with identity-based authentication and dynamic credentials.

Lesson Content

Secret sprawl is one of the most dangerous problems in cloud-native security. API keys in environment variables, database passwords in ConfigMaps, certificates that never rotate, and Vault tokens that need their own distribution mechanism. Each secret is a breach vector. Workload identity replaces most of these with cryptographic proof that needs no distribution.

Kubernetes Secrets: The Problem

Kubernetes Secrets are base64-encoded (not encrypted) by default. They are accessible to anyone with RBAC access to the namespace. They persist in etcd and must be encrypted at rest explicitly. They have no automatic rotation mechanism.

HashiCorp Vault Integration

Vault provides encrypted secret storage, dynamic credentials (database passwords generated on demand and automatically revoked), PKI certificates, and transit encryption. The key integration: workloads authenticate to Vault using their SPIFFE SVID instead of static Vault tokens.

Dynamic Secrets

# Instead of: static database password shared across all services
# Vault generates: unique short-lived credentials per service
# Each credential is automatically revoked after TTL expires

# Vault dynamic database credential:
vault read database/creds/readonly
# Key             Value
# lease_id        database/creds/readonly/abc123
# lease_duration  1h
# username        v-svc-readonly-xyz123  (unique per request!)
# password        A1b2C3-random-xyz      (auto-revoked in 1 hour)

Replacing Secrets with Workload Identity

The ultimate goal: eliminate static secrets entirely. Services authenticate with their SVID (automatic, short-lived, no distribution needed). Vault issues dynamic credentials based on verified identity. Certificates are managed by SPIRE (automatic rotation). The only secret that remains is the Vault unseal key — protected by cloud KMS.

Real-World Use Cases

• Eliminating static database passwords with Vault dynamic secrets
• SPIFFE-based Vault authentication (no token distribution)
• Certificate rotation for internal PKI
• Encrypting application data with Vault transit

Common Mistakes

• Storing Vault tokens as Kubernetes Secrets (replaces one secret problem with another)
• Not encrypting Kubernetes Secrets at rest in etcd
• Using static database passwords shared across all services
• Not implementing credential rotation — "it works fine" until the breach

Production Story

An e-commerce platform had 47 services sharing the same database password via environment variables. When the password was rotated, 12 services crashed because their cached connections used the old password. After switching to Vault dynamic credentials, each service got unique credentials with automatic rotation — zero-downtime rotations became routine.

Career Relevance

Secrets management is a critical skill for DevOps and platform engineering roles. Organizations need engineers who can eliminate secret sprawl and implement dynamic credential workflows.

Key Terms

Secret Sprawl

Uncontrolled proliferation of credentials across systems and configs

Dynamic Secrets

Credentials generated on-demand with automatic expiration

Vault

HashiCorp secret management platform

Transit Encryption

Vault engine for encrypting/decrypting data without storing it

Hands-On Labs

1. Integrate Vault with Kubernetes

   Deploy Vault and configure Kubernetes authentication.

   35 min - Intermediate

   • Deploy Vault in dev mode on Kubernetes
   • Enable Kubernetes auth backend
   • Create a policy for a service account
   • Retrieve secrets from a pod using the Vault agent

   View lab files on GitHub

2. Dynamic Secret Rotation

   Configure Vault to issue short-lived database credentials.

   30 min - Intermediate

   • Deploy PostgreSQL
   • Configure Vault database secrets engine
   • Generate dynamic credentials with 1-hour TTL
   • Verify automatic revocation after TTL expires

   View lab files on GitHub

Key Takeaways

• Secret sprawl is a top cloud-native security risk — every static secret is a breach vector
• Vault provides encrypted storage, dynamic credentials, and PKI — but needs its own auth
• SPIFFE SVIDs replace Vault tokens — workloads authenticate to Vault with their identity
• Dynamic secrets (short-lived, auto-revoked) are always better than static passwords
• The goal: zero static secrets in your infrastructure