The Mental Model
Data incidents are production incidents. A wrong dashboard can be as damaging as a down API when leaders use it to make decisions.
When a number is wrong, do not randomly edit SQL. Scope the issue, trace upstream, find the first bad layer, fix it, and write what prevented detection.
Dataset reference: ecommerce tables and grain assumptions.
Localize a revenue regression
After a deployment, revenue falls by 40% while order count stays at 100. Compare intermediate results at the same snapshot and time window.
| Check | Before | After |
|---|---|---|
| Completed order count | 100 | 100 |
| Gross amount | 10,000 | 10,000 |
| Refund total | 500 | 4,300 |
| Net revenue | 9,500 | 5,700 |
The first observed divergence is the refund total. Inspect its filters, units, join cardinality, and recent code changes. This table narrows the investigation; it does not prove a particular root cause. Preserve the failing query and source snapshot, correct the model, backfill affected partitions, and reconcile consumer totals.
Acceptance check: add a regression fixture that reproduces the actual defect, then confirm both the corrected result and unaffected date ranges.
Interactive Check
Question: Revenue drops 40% but order count is normal. What should you check first?
Reveal the answer
Check payment/refund amount logic, currency/unit conversion, filters on successful orders, and recent changes in models feeding the revenue metric.
Practice: Debug a Wrong Metric
Use a fake incident timeline to identify the most likely failing model.
Use the guided lab below to record your result, assumptions, and the check that would catch an incorrect result.