The Mental Model
Incremental models process only new or changed data to reduce cost. The hard part is correctness when data arrives late or historical logic changes.
Instead of rewriting an entire notebook every day, you add only today's pages. But if yesterday's page was corrected, you need a way to update it.
Tiny Example
We will use a small ecommerce dataset throughout the course. Think of these as the only tables in your first warehouse:
| Table | Grain | Example columns |
|---|---|---|
raw_orders | one row per order event | order_id, customer_id, amount, status, created_at |
raw_order_items | one row per item inside an order | order_id, product_id, quantity, item_price |
raw_customers | one row per customer | customer_id, email, country, created_at |
Interactive Check
Question: An order from Monday arrives in the source on Wednesday. What can go wrong in a naive incremental model?
Reveal the answer
The model may only process Wednesday rows and miss the Monday order because its event timestamp is old. Use a lookback window or update timestamp strategy.
Inline Practice Lab
This lab is intentionally small. You can solve it by reading the table, writing the SQL/YAML mentally, or pasting the snippet into any SQL scratchpad later.
-- Example starter table
select
order_id,
customer_id,
amount,
status,
created_at
from raw_orders;The goal is not tooling setup. The goal is learning the production habit: state the grain, clean one thing, test one assumption, and explain the downstream impact.
Self-Check Quiz
- What is the grain of the table you are building?
- Which downstream metric or dashboard would be wrong if this model broke?
- What test would catch the most likely beginner mistake here?