Data platform

Data pipeline: what it is, how an ETL pipeline differs, and what keeps one reliable

The stages every data pipeline has, the difference between a batch ETL pipeline and a streaming one, and the three things that keep it reliable.

  • Saudi technology company, in Riyadh since 1999 Interkey, Riyadh
  • ISO 9001:2015 and ISO/IEC 27001:2022 Quality and information security management
  • Implements and operates, after go-live too The same team before and after go-live

A data pipeline is a set of automated steps that moves data from where it is produced to where it is used. An ETL pipeline is one kind: it runs in batches and transforms data before loading it. A streaming pipeline moves each change as it happens. Every pipeline needs monitoring, a retry rule and a named owner.

The five stages of a data pipeline

Every pipeline passes data through the same five stages.

  1. Source.

    Where the data is produced: application databases, files that systems export, application interfaces (APIs) and devices.

  2. Capture.

    Reading the data out: everything on a schedule, only what changed since the last run, or the database's own change log, as capturing changes from the log explains.

  3. Processing.

    Cleaning, unifying, joining and calculating, when the data is reshaped for reports. When it is copied to another system as it is, this stage only checks it.

  4. Load.

    Writing to the destination: a data warehouse, a reporting database or another application. A stable key means a run repeated after a failure adds no duplicates.

  5. Use.

    Reports, dashboards, applications and people read the result. This is where a broken pipeline is usually noticed, so the aim is to notice it sooner.

A data pipeline, end to end

  1. Source

    databases, exported files, APIs

  2. Capture

    full read incremental read change log
  3. Processing

    clean unify join check
  4. Destination

    a data warehouse, a reporting database or another application

  5. Monitoring

    freshness, volume and errors at each stage

Monitoring is not a last step: it watches every stage before it.

Batch pipeline or streaming pipeline

The two common forms of a data pipeline
Batch pipeline (ETL)Streaming pipeline
When data movesOn a schedule: nightly, hourly, or at month endContinuously, each change soon after it happens
How fresh the destination isAs fresh as the last runA few seconds or minutes behind the source
Where data is reshapedBefore loading, in the pipelineUsually little or none; each change is copied as it is
What it suitsReports, analysis and a data warehouse read once a dayKeeping two operational systems in step, and alerts that cannot wait
What a failure looks likeA run fails; the morning report shows yesterday's figuresChanges pile up; the destination falls further behind until the cause is fixed

Where iblync fits

When the pipeline connects operational systems

iblync, Interkey's sync and migration platform, runs the streaming kind between different systems: continuous real-time sync, with transfer speed and resource use under control. If the target stops, changes wait in a queue and are processed when it returns.

Data pipeline architecture in practice

Most data pipeline architectures combine three patterns. An ETL pipeline (extract, transform, load) reshapes data before it reaches a warehouse. ELT loads the raw data first and transforms it inside the warehouse; ETL or ELT sets out when each fits. Change data capture reads each change from the source database's log and sends it on, which is how a streaming pipeline usually starts.

The choice is made per data set, not once for the whole company. Finance figures read monthly can run in batches; stock levels shared between two systems cannot wait for the night.

Why pipelines fail

Six causes behind most broken pipelines.

  • The source schema changes: a column is renamed, added or given a new type, and the next stage rejects records or loads empty values.
  • Data arrives late: a source export runs late, and the report shows yesterday's figures without saying so.
  • Duplicates: a run repeated after a failure loads the same records twice, because the load has no stable key.
  • The destination fills up: storage runs out and every write after that fails.
  • Credentials expire: a password or access token the pipeline uses runs out, and the night run fails.
  • No one owns the pipeline: alerts go to a shared mailbox, and the failure is found when a manager questions a number.

Three things that keep a pipeline reliable

02

A retry rule and a queue

Decide what happens when a stage fails: how many retries, and where changes wait meanwhile. iblync keeps changes in a queue when the target stops and processes them when it returns.

03

A named owner

One person or team per pipeline receives its alerts, approves changes at the source and decides whether to rerun. Without one, every alert is someone else's.

What to watch at each stage

Pipeline monitoring starts with three questions per stage. Did data arrive on time? Did the expected number of records come through? Were any records rejected, and why? Count records in and out of every run, keep rejected records in their own table with the reason, and record when each table last changed.

Pipelines that carry logs and metrics rather than business data have their own needs, such as filtering and masking before the data leaves: see telemetry pipelines for logs and metrics.

Questions about data pipelines

What is a data pipeline?

A data pipeline is a set of automated steps that moves data from where it is produced to where it is used. It reads from sources such as application databases and exported files, processes the data where needed, and writes it to a destination such as a data warehouse or another application. Batch ETL pipelines and streaming pipelines are its two common forms.

What is an ETL pipeline?

An ETL pipeline is a data pipeline that runs in batches and transforms data before loading it. On a schedule, often nightly, it extracts data from the sources, cleans and reshapes it, then loads the result into a target such as a data warehouse. It suits reports that are read once a day. What is ETL? walks through each step.

What is the difference between a data pipeline and ETL?

A data pipeline is the general name, and ETL is one kind of it. Any automated flow of data from a source to a destination is a pipeline, whether it runs in batches or as a stream, and whether it reshapes the data or copies it unchanged. ETL names one pattern: extract in batches, transform, then load. ELT, which transforms inside the target, is another; see ETL vs ELT.

How do you monitor a data pipeline?

You monitor a data pipeline by measuring freshness, volume and errors at each stage, and alerting when one of them leaves its usual range. Freshness shows when data last arrived, volume shows whether the expected records came through, and errors show failed runs and rejected records. Data observability explains how these checks are set up across tables and pipelines.

What happens when the destination is down?

A well-built pipeline keeps the changes in a queue and processes them when the destination returns. Without a queue, changes are lost or the whole pipeline stops, and someone reloads data by hand. Ask any supplier where changes wait during an outage and how the pipeline catches up.

Next step

Ask an engineer one question

Tell us where your data starts and where it must arrive, and ask one question about your pipeline.

Required field

By sending this form, you agree to our Privacy Notice.

What happens next

  1. We read your request
  2. An engineer contacts you
  3. You get an estimate or a demo

Or directly

+966-11-2180999 [email protected]

Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia

or What is ETL?

Published by Interkey. Last updated . Interkey is registered in Riyadh, Saudi Arabia under commercial registration 1010156897.