Reprocessing an entire table every night works fine until the table gets big enough that “every night” starts bleeding into the next morning. Change data capture vs batch ETL is the choice between reprocessing everything on a schedule and streaming only what actually changed.
What batch ETL does by default
A traditional batch job reads a full table, or a large slice of one, transforms it, and loads the result on a fixed schedule. It’s simple to reason about and easy to rerun. The cost scales with total data volume, not with how much actually changed since the last run, which becomes a real problem once a table grows large relative to how often it needs refreshing.
What change data capture does instead
CDC reads a database’s transaction log directly and streams individual row-level changes, inserts, updates, deletes, as they happen. Instead of asking “what does the whole table look like now,” it asks “what changed since I last checked,” and only moves that. For a large table with a small fraction of rows changing per day, this is a fundamentally smaller amount of work.
A real example: where batch ETL is the right fit
The retail ETL pipeline validates raw exports at a data-quality gate on a batch basis before loading into the warehouse. At the scale of a retail export refreshed on a schedule, batch processing is the simpler, more maintainable choice. Nothing about that workload demands sub-minute freshness, so the added complexity of CDC wouldn’t pay for itself here.
Where CDC actually earns its complexity
CDC makes sense when a downstream system genuinely needs near-real-time visibility into changes, and reprocessing the full source table on every run would be too slow or too expensive to keep up. It’s a common pattern for syncing a production database into a warehouse without placing repeated heavy read load on the production system itself.
The real tradeoff: complexity and failure modes
CDC introduces its own operational surface: a connector reading the transaction log, ordering guarantees, and handling schema changes on the source side gracefully. A batch job that fails can usually just be rerun from scratch. A CDC stream that falls behind or misses an event needs its own recovery story, which is real added complexity that has to be maintained, not a one-time setup cost.
A quick checklist
- Does the downstream system actually need near-real-time data, or is a scheduled batch refresh genuinely sufficient?
- Is the source table large enough, relative to its change rate, that full reprocessing has become a real bottleneck?
- Are you prepared to handle CDC’s own failure modes, like connector lag or schema drift on the source?
- Would the operational overhead of CDC actually pay off here, or would it just add complexity for a workload batch already handles fine?
FAQ
Is CDC always better than batch ETL for large tables?
Not automatically. Table size alone doesn’t justify CDC, what matters is the ratio of changes to total data, and whether downstream systems genuinely need low-latency updates.
Can CDC and batch ETL coexist in the same pipeline?
Yes, commonly. Some sources are CDC-streamed for freshness while others, especially smaller or less time-sensitive ones, stay on a simpler batch schedule.
Does CDC replace the need for data-quality tests?
No. CDC changes how data arrives, not whether it’s valid once it does. The same validation discipline used in batch pipelines still applies to CDC-streamed data.
