Module 13 of 16

Observability & Security Monitoring

OpenTelemetry, audit logging, distributed tracing, and security telemetry

3 hours2 labsFree

Start here

Learning objectives

  • Build security-focused observability with OpenTelemetry
  • Configure Kubernetes audit logging
  • Correlate security events across services
  • Design dashboards for security posture monitoring
SECURITY TELEMETRY PIPELINEWorkloads + K8sOpenTelemetryCollectortraces + metrics + logsPrometheusLoki (Logs)GrafanaAlertManagerKey Security MetricsFailed auth attemptsSVID rotation stallsPolicy violationsFalco alerts rateAnomalous traffic

Security without visibility is flying blind. You need to know: who accessed what, when, whether SVID rotation is healthy, how many policy violations occurred, and whether any runtime alerts were triggered. Security observability connects all your security controls into a unified monitoring layer.

Kubernetes Audit Logging

Audit logs capture every API request: who made it, what was requested, and what was the result. Critical for compliance (SOC 2, PCI-DSS) and incident investigation.

# Audit policy: log all authentication failures and secret access
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - level: RequestResponse
    users: ["system:anonymous"]
  - level: Metadata
    verbs: ["create", "delete", "patch"]

OpenTelemetry for Security

OpenTelemetry collects traces, metrics, and logs from all services. For security, this means: correlating authentication events with service behavior, tracking SVID rotation health, monitoring policy decision latency, and tracing request flows across mTLS boundaries.

Security Dashboards

Essential Grafana dashboards: authentication failures over time, SVID rotation success rate, policy violations by namespace, runtime alerts (Falco/Tetragon), certificate expiry countdown, and API server audit log analysis.

Real world

Where this shows up

  • Security dashboards for SOC teams
  • Kubernetes audit log analysis for compliance
  • Correlating Falco alerts with application traces
  • SIEM integration for centralized security monitoring

Key terms

Vocabulary used in this module

Audit Log

Kubernetes record of every API request — who, what, when, result

OpenTelemetry

CNCF observability framework for traces, metrics, and logs

SIEM

Security Information and Event Management — centralized security log analysis

Labs

Hands-on labs

35 minIntermediate

Build Security Dashboards

Create Grafana dashboards for security posture monitoring.

  1. Deploy Prometheus, Loki, and Grafana
  2. Configure security metric scraping (SPIRE, Falco, OPA)
  3. Build dashboards for auth failures, SVID rotation, policy violations
  4. Set up alerts for security-critical thresholds
View lab on GitHub
25 minIntermediate

Kubernetes Audit Log Analysis

Enable and analyze audit logs for security investigation.

  1. Configure API server audit policy
  2. Generate events (access secrets, create roles)
  3. Search audit logs for suspicious patterns
  4. Correlate audit events with application traces
View lab on GitHub

Recap

Key takeaways

  • Security observability connects all controls into unified monitoring
  • Kubernetes audit logs are essential for compliance and incident investigation
  • OpenTelemetry provides the collection layer for security telemetry
  • Dashboard the security posture: auth failures, SVID health, policy violations, runtime alerts
  • Alert on security metrics — do not wait for incidents to discover monitoring gaps

Related resources

Keep learning across CodersSecret