Monitoring
Measure freshness, volume and errors at every stage, and alert when one leaves its usual range. Watching tables and pipelines with Datadog shows how.
Data platform
The stages every data pipeline has, the difference between a batch ETL pipeline and a streaming one, and the three things that keep it reliable.
Explainer Published 5 min read
A data pipeline is a set of automated steps that moves data from where it is produced to where it is used. An ETL pipeline is one kind: it runs in batches and transforms data before loading it. A streaming pipeline moves each change as it happens. Every pipeline needs monitoring, a retry rule and a named owner.
Every pipeline passes data through the same five stages.
Where the data is produced: application databases, files that systems export, application interfaces (APIs) and devices.
Reading the data out: everything on a schedule, only what changed since the last run, or the database's own change log, as capturing changes from the log explains.
Cleaning, unifying, joining and calculating, when the data is reshaped for reports. When it is copied to another system as it is, this stage only checks it.
Writing to the destination: a data warehouse, a reporting database or another application. A stable key means a run repeated after a failure adds no duplicates.
Reports, dashboards, applications and people read the result. This is where a broken pipeline is usually noticed, so the aim is to notice it sooner.
Source
databases, exported files, APIs
Capture
Processing
Destination
a data warehouse, a reporting database or another application
Monitoring
freshness, volume and errors at each stage
Monitoring is not a last step: it watches every stage before it.
| Batch pipeline (ETL) | Streaming pipeline | |
|---|---|---|
| When data moves | On a schedule: nightly, hourly, or at month end | Continuously, each change soon after it happens |
| How fresh the destination is | As fresh as the last run | A few seconds or minutes behind the source |
| Where data is reshaped | Before loading, in the pipeline | Usually little or none; each change is copied as it is |
| What it suits | Reports, analysis and a data warehouse read once a day | Keeping two operational systems in step, and alerts that cannot wait |
| What a failure looks like | A run fails; the morning report shows yesterday's figures | Changes pile up; the destination falls further behind until the cause is fixed |
When the pipeline connects operational systems
iblync, Interkey's sync and migration platform, runs the streaming kind between different systems: continuous real-time sync, with transfer speed and resource use under control. If the target stops, changes wait in a queue and are processed when it returns.
Most data pipeline architectures combine three patterns. An ETL pipeline (extract, transform, load) reshapes data before it reaches a warehouse. ELT loads the raw data first and transforms it inside the warehouse; ETL or ELT sets out when each fits. Change data capture reads each change from the source database's log and sends it on, which is how a streaming pipeline usually starts.
The choice is made per data set, not once for the whole company. Finance figures read monthly can run in batches; stock levels shared between two systems cannot wait for the night.
Six causes behind most broken pipelines.
Measure freshness, volume and errors at every stage, and alert when one leaves its usual range. Watching tables and pipelines with Datadog shows how.
Decide what happens when a stage fails: how many retries, and where changes wait meanwhile. iblync keeps changes in a queue when the target stops and processes them when it returns.
One person or team per pipeline receives its alerts, approves changes at the source and decides whether to rerun. Without one, every alert is someone else's.
Pipeline monitoring starts with three questions per stage. Did data arrive on time? Did the expected number of records come through? Were any records rejected, and why? Count records in and out of every run, keep rejected records in their own table with the reason, and record when each table last changed.
Pipelines that carry logs and metrics rather than business data have their own needs, such as filtering and masking before the data leaves: see telemetry pipelines for logs and metrics.
A data pipeline is a set of automated steps that moves data from where it is produced to where it is used. It reads from sources such as application databases and exported files, processes the data where needed, and writes it to a destination such as a data warehouse or another application. Batch ETL pipelines and streaming pipelines are its two common forms.
An ETL pipeline is a data pipeline that runs in batches and transforms data before loading it. On a schedule, often nightly, it extracts data from the sources, cleans and reshapes it, then loads the result into a target such as a data warehouse. It suits reports that are read once a day. What is ETL? walks through each step.
A data pipeline is the general name, and ETL is one kind of it. Any automated flow of data from a source to a destination is a pipeline, whether it runs in batches or as a stream, and whether it reshapes the data or copies it unchanged. ETL names one pattern: extract in batches, transform, then load. ELT, which transforms inside the target, is another; see ETL vs ELT.
You monitor a data pipeline by measuring freshness, volume and errors at each stage, and alerting when one of them leaves its usual range. Freshness shows when data last arrived, volume shows whether the expected records came through, and errors show failed runs and rejected records. Data observability explains how these checks are set up across tables and pipelines.
A well-built pipeline keeps the changes in a queue and processes them when the destination returns. Without a queue, changes are lost or the whole pipeline stops, and someone reloads data by hand. Ask any supplier where changes wait during an outage and how the pipeline catches up.
Next
Next step
Tell us where your data starts and where it must arrive, and ask one question about your pipeline.
What happens next
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia
or What is ETL?