Module 13: Observability & Security Monitoring

Learning Objectives

• Build security-focused observability with OpenTelemetry
• Configure Kubernetes audit logging
• Correlate security events across services
• Design dashboards for security posture monitoring

Why This Matters

You have deployed identity, encryption, policy, and runtime security. But are they working? Observability tells you. Without it, you only discover problems after the breach.

Lesson Content

Security without visibility is flying blind. You need to know: who accessed what, when, whether SVID rotation is healthy, how many policy violations occurred, and whether any runtime alerts were triggered. Security observability connects all your security controls into a unified monitoring layer.

Kubernetes Audit Logging

Audit logs capture every API request: who made it, what was requested, and what was the result. Critical for compliance (SOC 2, PCI-DSS) and incident investigation.

# Audit policy: log all authentication failures and secret access
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - level: RequestResponse
    users: ["system:anonymous"]
  - level: Metadata
    verbs: ["create", "delete", "patch"]

OpenTelemetry for Security

OpenTelemetry collects traces, metrics, and logs from all services. For security, this means: correlating authentication events with service behavior, tracking SVID rotation health, monitoring policy decision latency, and tracing request flows across mTLS boundaries.

Security Dashboards

Essential Grafana dashboards: authentication failures over time, SVID rotation success rate, policy violations by namespace, runtime alerts (Falco/Tetragon), certificate expiry countdown, and API server audit log analysis.

Real-World Use Cases

• Security dashboards for SOC teams
• Kubernetes audit log analysis for compliance
• Correlating Falco alerts with application traces
• SIEM integration for centralized security monitoring

Production Story

A security team discovered a privilege escalation attempt only after reviewing audit logs 2 weeks later. After deploying real-time alerting on audit events, the same pattern was caught in under 5 minutes.

Career Relevance

Security observability bridges the gap between security engineering and SRE. Engineers who can build security dashboards and alerting are valuable across both disciplines.

Key Terms

Audit Log

Kubernetes record of every API request — who, what, when, result

OpenTelemetry

CNCF observability framework for traces, metrics, and logs

SIEM

Security Information and Event Management — centralized security log analysis

Hands-On Labs

1. Build Security Dashboards

   Create Grafana dashboards for security posture monitoring.

   35 min - Intermediate

   • Deploy Prometheus, Loki, and Grafana
   • Configure security metric scraping (SPIRE, Falco, OPA)
   • Build dashboards for auth failures, SVID rotation, policy violations
   • Set up alerts for security-critical thresholds

   View lab files on GitHub

2. Kubernetes Audit Log Analysis

   Enable and analyze audit logs for security investigation.

   25 min - Intermediate

   • Configure API server audit policy
   • Generate events (access secrets, create roles)
   • Search audit logs for suspicious patterns
   • Correlate audit events with application traces

   View lab files on GitHub

Key Takeaways

• Security observability connects all controls into unified monitoring
• Kubernetes audit logs are essential for compliance and incident investigation
• OpenTelemetry provides the collection layer for security telemetry
• Dashboard the security posture: auth failures, SVID health, policy violations, runtime alerts
• Alert on security metrics — do not wait for incidents to discover monitoring gaps