A source system changes a field name, and three pipelines downstream break without warning. Data contracts in data engineering exist to catch that before it happens, not after.
What a data contract actually is
A data contract is an explicit agreement between whoever produces data and whoever consumes it: field names, types, allowed values, and update frequency, written down and enforced rather than assumed. Without one, a schema change on the producer’s side is invisible until something downstream fails.
Why this connects to data-quality testing
The retail ETL pipeline validates raw exports at a data-quality gate before anything reaches the warehouse. That gate is effectively enforcing an informal contract: these fields must exist, these values must fall in range. A formal data contract makes that same expectation explicit and shared with whoever produces the data upstream, rather than leaving it as a check that only the downstream pipeline knows about.
Where automated tests act as an informal contract
The retail analytics warehouse and Olist analytics engineering project both run dozens of automated dbt tests checking uniqueness, referential integrity, and accepted values. Those tests catch a broken contract after the fact, when a build fails. A true data contract aims to catch the same violation earlier, ideally before a producer even ships the breaking change.
Why this matters more as a team grows
On a small project, one person controls both the source and the pipeline, so schema drift gets caught quickly just by proximity. Data contracts start to matter once producers and consumers are different teams, since nobody downstream automatically knows a field changed until a report looks wrong or a test fails.
What a basic contract needs to cover
- A defined schema: field names, types, and whether a field can be null.
- Expected value ranges or allowed categories, not just the data type.
- An update or versioning policy for what happens when the schema needs to change.
- Ownership: who to notify before a breaking change ships.
A quick checklist
- If an upstream system changed a field tomorrow, would you find out before or after something broke?
- Are your data-quality tests documenting an implicit contract, or is that contract written down anywhere?
- Does anyone producing data for your pipeline know what your pipeline actually expects?
- Is there a process for a schema change to be reviewed before it ships, not just detected after?
FAQ
Do small projects need formal data contracts?
Not usually. Automated data-quality tests, like the ones in the retail and Olist warehouse projects, cover most of the same ground at a smaller scale.
Is a data contract the same as a schema definition?
A schema definition is part of it, but a contract typically also covers value ranges, update frequency, and ownership, not just field types.
Who should own a data contract, the producer or the consumer?
Ideally both agree to it jointly, since a contract only works if the producer actually commits to not breaking it silently.
