Module 10: Runtime Security & Threat Detection

Learning Objectives

• Understand runtime threat categories in Kubernetes
• Deploy Falco for syscall-based threat detection
• Use Tetragon for eBPF-based enforcement
• Build incident response procedures for runtime events

Why This Matters

All the identity, encryption, and policy controls in earlier modules prevent unauthorized access. But what about an authorized workload that gets compromised? Runtime security is the last line of defense — it detects and responds to threats that bypass all other controls.

Lesson Content

Identity prevents unauthorized access. Network policies restrict traffic. But what about threats INSIDE an authorized workload? A compromised container running a cryptominer, an attacker spawning a shell, malware reading sensitive files — these are runtime threats that require real-time detection and response.

Falco: Syscall-Based Detection

Falco is a CNCF graduated project that monitors Linux syscalls and alerts on suspicious behavior. It runs as a DaemonSet and watches every container on the node.

# Falco rule: detect shell spawned in container
- rule: Terminal shell in container
  desc: A shell was spawned in a container
  condition: >
    spawned_process and container and
    proc.name in (bash, sh, zsh, dash)
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name
     image=%container.image.repository
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING

Tetragon: eBPF Runtime Enforcement

Tetragon goes beyond detection — it can block malicious actions in real-time using eBPF. While Falco alerts you to a shell spawn, Tetragon can prevent the shell from executing.

eBPF: The Foundation

eBPF (extended Berkeley Packet Filter) runs sandboxed programs in the Linux kernel without kernel modules. It enables runtime security tools to observe and enforce at the kernel level — with near-zero performance overhead.

Incident Response

Detection without response is just monitoring. The response pipeline should: alert the security team (Slack/PagerDuty), capture forensic data (pod logs, network connections), isolate the pod (NetworkPolicy deny-all), preserve evidence (do not delete the pod), and investigate root cause.

Real-World Use Cases

• Detecting cryptominers in production containers
• Container escape attempt alerting
• Runtime threat detection for compliance (PCI-DSS)
• Automated incident response for security events

Common Mistakes

• Running Falco without custom rules (defaults miss many threats)
• Not integrating alerts with incident response workflows
• Using Tetragon enforcement rules without thorough testing (can block legitimate operations)
• Not preserving forensic evidence when responding to incidents

Production Story

A production cluster was running a cryptominer for 3 weeks before anyone noticed the CPU spike. After deploying Falco, the same behavior was detected within 30 seconds and the security team was alerted via PagerDuty. The compromised container was isolated automatically.

Career Relevance

Runtime security is the fastest-growing area of Kubernetes security. Falco and eBPF skills are increasingly listed in security engineering and SRE job descriptions.

Key Terms

Falco

CNCF runtime security tool that monitors syscalls for suspicious behavior

Tetragon

Cilium eBPF-based runtime enforcement engine

eBPF

Extended Berkeley Packet Filter — runs programs in the kernel sandbox

Syscall

System call — interface between user programs and the kernel

Hands-On Labs

1. Detect Container Escape Attempts with Falco

   Deploy Falco and trigger security alerts.

   30 min - Intermediate

   • Install Falco as a DaemonSet
   • Spawn a shell inside a container
   • Read /etc/shadow from inside a container
   • Observe Falco alerts for both events

   View lab files on GitHub

2. Runtime Enforcement with Tetragon

   Block malicious actions at the kernel level.

   30 min - Advanced

   • Install Tetragon
   • Create a TracingPolicy that blocks shell execution
   • Attempt to spawn a shell — observe block
   • View Tetragon event logs

   View lab files on GitHub

Key Takeaways

• Runtime security detects threats INSIDE authorized workloads — the last line of defense
• Falco monitors syscalls and alerts on suspicious behavior (CNCF graduated)
• Tetragon uses eBPF to BLOCK malicious actions, not just detect
• eBPF enables kernel-level observability with near-zero overhead
• Detection without response is just monitoring — build the full incident pipeline