Module 14 of 16

Data Incidents and Debugging

Debug wrong revenue, stale data, broken joins, and schema drift like an engineer.

Updated

110 minutes1 exercisesFree

Start here

Learning objectives

  • Classify common data incidents
  • Use tests and lineage during debugging
  • Write a useful data incident review
Data Incidents and Debugging Follow the arrows. Each box is one idea you will practice in this module. Alert step 1 Scope step 2 Trace step 3 Fix step 4 Review step 5 Production analytics engineering turns raw records into governed, trusted business meaning.

The Mental Model

Data incidents are production incidents. A wrong dashboard can be as damaging as a down API when leaders use it to make decisions.

When a number is wrong, do not randomly edit SQL. Scope the issue, trace upstream, find the first bad layer, fix it, and write what prevented detection.

Dataset reference: ecommerce tables and grain assumptions.

Localize a revenue regression

After a deployment, revenue falls by 40% while order count stays at 100. Compare intermediate results at the same snapshot and time window.

CheckBeforeAfter
Completed order count100100
Gross amount10,00010,000
Refund total5004,300
Net revenue9,5005,700

The first observed divergence is the refund total. Inspect its filters, units, join cardinality, and recent code changes. This table narrows the investigation; it does not prove a particular root cause. Preserve the failing query and source snapshot, correct the model, backfill affected partitions, and reconcile consumer totals.

Acceptance check: add a regression fixture that reproduces the actual defect, then confirm both the corrected result and unaffected date ranges.

Interactive Check

Question: Revenue drops 40% but order count is normal. What should you check first?

Reveal the answer

Check payment/refund amount logic, currency/unit conversion, filters on successful orders, and recent changes in models feeding the revenue metric.

Practice: Debug a Wrong Metric

Use a fake incident timeline to identify the most likely failing model.

Use the guided lab below to record your result, assumptions, and the check that would catch an incorrect result.

Production notes

Keep these close

  • Maintain a data incident template: symptom, impact, first bad layer, detection gap, fix, prevention.

Common mistakes

What usually breaks

  • Fixing the dashboard instead of the model
  • Skipping incident review after numbers recover
  • Not notifying metric owners and consumers

Key terms

Vocabulary used in this module

Data incident

A reliability event where data is wrong, late, incomplete, or misleading.

Blast radius

The set of downstream users, models, or metrics affected by a change or failure.

Exercises

Practice inside the lesson

30-45 minutesIntermediate

Debug a Wrong Metric

Use a fake incident timeline to identify the most likely failing model.

  1. Read the symptoms
  2. List affected metrics
  3. Trace upstream models
  4. Pick the first layer where values diverge
  5. Write one test that would have caught it

Expected evidence

A short answer, SQL/YAML snippet, or lineage map that can live directly in the course page notes.

Recap

Key takeaways

  • Data debugging needs scope, lineage, and tests
  • Incidents should produce prevention work
  • Wrong data is a reliability problem

Related resources

Keep learning across CodersSecret