Notes on data engineering, AI and building things end to end.
Two records for the same customer, entered slightly differently, don’t announce themselves as duplicates. Here’s how data deduplication techniques actually handle that, with a real project example.
Read more →
A single random seed can make a model look better or worse than it really is. Here’s why random seed reproducibility matters more than one lucky training run.
Read more →
Supervised vs unsupervised learning comes down to one question: does your data already have the answer attached? Here’s how each works, with real project examples.
Read more →
A model that looked good offline can still fail in production. Here’s blue-green vs canary deployment, two ways to roll out a new model version safely.
Read more →
A test catches a known rule violation. It won’t catch a table that quietly stopped updating. Here’s data observability vs data quality testing, and why both matter.
Read more →
A wrong number on a dashboard raises a question: where did it come from? Here’s what data lineage in data engineering actually tracks, and how dbt provides it almost for free.
Read more →
Reprocessing a full table works until it doesn’t. Here’s change data capture vs batch ETL, when each is the right fit, and a real project example of batch done well.
Read more →
A running total or rank within a group used to mean a slow, hard-to-read subquery. Here’s window functions vs subqueries in SQL, and when each actually fits.
Read more →
When a dimension attribute changes, do you overwrite it or keep history? Here’s slowly changing dimensions type 1 vs type 2, and why the choice affects historical reports.
Read more →
A failed pipeline gets rerun, and now rows exist twice. Here’s why idempotent data pipelines matter and the patterns that actually prevent duplication.
Read more →