A data pipeline that runs once on your laptop is a script. A pipeline that has to run every night, in the right order, and tell someone when it fails needs an orchestrator.
What an orchestrator does
An orchestrator schedules pipeline tasks, runs them in the right order, retries the ones that fail, and reports what happened. It does not usually do the heavy data processing itself. It tells other systems, such as a warehouse, a transformation tool or a script, when to run and in what sequence. Apache Airflow, Dagster and Prefect are well-known examples.
Dependencies and the DAG
Pipelines are usually modeled as a directed acyclic graph, or DAG: tasks are nodes, and an arrow means one task must finish before another starts. The load step waits for the extract step, and the report refresh waits for the load. Because the graph has no cycles, the orchestrator always knows a valid order, and it can run independent branches at the same time.
Why cron is not enough
A cron job can start a script at a set time, and for a single small task that is often enough. It does not know that task B depends on task A, it does not retry on failure, and it gives you no history of past runs. Once a pipeline has several dependent steps, those gaps turn into missed data and mornings spent finding out what broke.
Retries and backfills
Two features make an orchestrator worth the effort. Retries handle temporary failures, such as a source system timing out, without a human stepping in. Backfills re-run the pipeline for past dates, for example after fixing a bug in the logic. Both are only safe if each task can run twice without creating duplicates, which is the property described in idempotent data pipelines. How much each run loads is a separate design choice, covered in incremental load vs full refresh.
Visibility is part of the job
A good orchestrator shows you which tasks ran, how long they took, and where a run failed, and it can alert you when something breaks. That run history is a base for the checks described in data observability, and the task graph itself is a rough map of data lineage.
Keep the orchestrator thin
A common mistake is to put the actual data processing inside the orchestrator’s own workers. That makes the scheduler a bottleneck and the logic harder to test. A cleaner pattern is for tasks to trigger work elsewhere, in the warehouse or a separate job, so the orchestrator stays focused on order, timing and status. Where the transformation happens is the question behind ETL vs ELT.
When you do not need one yet
For one or two independent jobs, a scheduled script with logging and an alert on failure is usually enough, and an orchestrator adds setup and upkeep you may not need. The signal to adopt one is when you find yourself manually tracking dependencies, rerunning failed jobs by hand, or asking what ran last night.
A quick checklist
- Do tasks depend on each other, or are they independent?
- Can every task run twice without duplicating data?
- Do you get an alert when a run fails?
- Can you re-run a past date range without editing code?
- Does the orchestrator trigger work, or is it doing the processing itself?
If you are planning a pipeline and want help choosing the setup, see my ETL and ELT engineering service.
FAQ
What is data pipeline orchestration?
It is the scheduling and coordination of pipeline tasks: running them in order, handling retries, and reporting status, usually with a tool such as Airflow, Dagster or Prefect.
Is Airflow an ETL tool?
Not exactly. Airflow is an orchestrator. It schedules and sequences work, while the extraction and transformation are done by the code or systems it triggers.
Do I need an orchestrator for a small project?
Often not. A scheduled script with logging and failure alerts can be enough until you have several dependent steps or need reliable reruns.
