Case study · Data Engineering · Forward Deployed
Data Incident Casebook
Three reproducible data incidents worked like investigations: symptom, hypotheses, evidence, root cause, fix and a guard that stays.
The problem
The worst data incidents raise no error: a report quietly loses rows, a join slowly stops matching, revenue drifts above the ledger. Fixing them is easy once you know where the data went wrong. Finding that place is the hard part.
Constraints
- No client data: every incident is rebuilt on generated data that behaves the same way
- Every number in a write-up must come from code, so the story can be re-run and checked
- Each fix has to leave a guard behind, so the same failure breaks a build next time instead of a report
Architecture and decisions
Click a box to see why it was chosen.
Why · Hypotheses
Cheapest checks first: did the source change, did a step drop rows, did a key change? Rejected hypotheses stay in the file to show what was ruled out.
Results
- incidents: join loss, key format change, duplicate events
- 3
- incidents: join loss, key format change, duplicate events
- customer match rate restored in incident 02
- 57% → 98%
- customer match rate restored in incident 02
- revenue gap vs the ledger closed in incident 03
- +15% → 0%
- revenue gap vs the ledger closed in incident 03
- reusable guard checks, 15 tests
- 5
- reusable guard checks, 15 tests
What I would do next
- 01Incident 04: a timezone change that moves late-night orders to the next day
- 02Ship the guards as dbt tests so they run on Snowflake and BigQuery