Observability & Debugging

Observability is what separates a system you can debug from one you cannot. In a monolith, debugging means reading a stack trace. In a distributed system, debugging means reconstructing a request across many services from telemetry alone. If your observability is poor, your incidents are unrecoverable. If it is good, the system tells you where it broke.

The Three Pillars (and the Fourth)

• Metrics: numeric time-series. Cheap to store, cheap to query, great for alerting and dashboards. Prometheus is the open-source standard.
• Logs: discrete events with timestamps and structure. Expensive to store at scale; great for debugging known issues.
• Traces: per-request causality across services. Heavy to capture and store; essential for debugging latency.
• Profiles (the modern fourth): CPU and allocation samples per service over time (Pyroscope, Parca). Catches what metrics aggregate away.

OpenTelemetry — The Standard

OpenTelemetry (OTel) is the CNCF project that unifies instrumentation. One SDK in your application emits all three signals; an OTel Collector batches, samples, and ships them to whichever backends you choose. The vendor lock-in problem of older instrumentation libraries is solved.

Production pattern: every service uses the OTel SDK; an OTel Collector runs as a DaemonSet on Kubernetes; the Collector forwards traces to Jaeger/Tempo, metrics to Prometheus, logs to Loki or Elasticsearch. Grafana is the unified UI on top.

Distributed Tracing

A trace is a tree of spans, where each span represents a unit of work (an HTTP call, a database query, a message handler). The root span is the user request; child spans are everything it triggered. Spans carry a trace ID (propagated across service boundaries via the W3C Trace Context header) and a span ID (parent reference).

Tracing answers questions metrics cannot: why is p99 of request type X high? A trace shows you exactly which downstream service contributed the latency. Why does this rare error happen? The trace shows the full causal chain. Where is this request actually going? The trace reveals architectural surprises (services calling services you forgot existed).

Sampling

Tracing every request is expensive at scale. Sampling reduces volume:

• Head-based sampling: decide at the start of the trace whether to keep it (e.g. 1% of all requests). Simple; misses error traces.
• Tail-based sampling: collect everything, decide after the trace completes (e.g. keep all error traces and a sample of success traces). Better visibility; harder to operate.
• Adaptive / hybrid: head-sample at modest rate; force-sample known interesting paths (errors, slow requests, specific endpoints).

The Four Golden Signals

From the Google SRE book, the four metrics every service should emit:

1. Latency: time to serve a request. Track p50, p95, p99; alert on p99.
2. Traffic: requests per second. Sudden change is a signal even if everything else looks fine.
3. Errors: failed requests. Express as a rate (errors per second) or a ratio (error rate over total RPS).
4. Saturation: how full the system is. CPU, memory, queue depth, connection pool utilisation. Saturation precedes other failures.

Every dashboard, every alert, every SLO connects back to these four. If you only emit four metrics per service, emit these.

Structured Logging and Correlation IDs

Logs are useful when they are queryable. That means structured (JSON or key=value) and correlated. Every log line should include the trace ID so you can filter by request and the user/tenant ID so you can debug per-user issues.

The minimum log line for a service: timestamp, level, service, trace_id, span_id, user_id, message, ...fields. Anything less and your logs are unsearchable at scale.

SLO and Error Budget

Service Level Objectives (SLOs) translate the four golden signals into commitments. “p99 latency < 300ms over 30 days” or “99.9% of requests return 2xx/3xx over 30 days”. The error budget is the difference between 100% and the SLO — the amount of failure you have permission to spend on risky changes, deploys, or experiments.

Operating with explicit SLOs and error budgets is the discipline of the modern SRE function. The pattern: when error budget is healthy, you ship features fast; when it is exhausted, you stop shipping and stabilise.

Debugging Distributed Systems

The flow that works in practice:

1. Alert fires ⇒ identify the affected service from dashboard.
2. Check the four golden signals for that service.
3. Pick a representative failed trace; walk it span by span; identify where latency or error appears.
4. If it is a downstream service, recurse: open that service's dashboard, repeat.
5. If it is in the service itself, jump to logs filtered by that trace ID.
6. If logs do not show the cause, jump to profiles.

That's the loop. Every minute saved in this loop is a minute off MTTR.

Distributed Request Trace Timeline

Self-Check Quiz

1. Why are metrics, logs, and traces complementary rather than redundant? (Answer: metrics tell you something is wrong; traces tell you which service; logs tell you why. Each answers a different question at a different cost.)
2. You have head-based sampling at 1%. Errors get under-represented. What is the fix? (Answer: tail-based sampling — sample after the trace completes, force-include error traces. Or hybrid head+force-on-error.)
3. Your dashboard shows error rate at 0.1% — within SLO. Customer support says many users complain. What is happening? (Answer: averages hide tail. Your 0.1% may be concentrated on one tenant or one feature. Slice metrics by user/tenant/feature, not just service-level.)
4. What are the four golden signals and why does saturation matter? (Answer: latency, traffic, errors, saturation. Saturation precedes the other three failing — the queue fills before latency climbs before errors fire.)

For runtime-detection observability the Runtime Security cheatsheet covers Falco/Tetragon eBPF telemetry alongside the application-layer signals.