Datadog · Data observability
Data observability: catch stale tables and broken pipelines before the dashboard is wrong
Data observability tells you when the data behind a report has stopped being true: a table that did not refresh, a load that brought half the usual rows, a column that suddenly fills with blanks, a job that failed at 3 a.m. This guide explains the problem in plain words, the ways teams catch it, and where Datadog Data Observability fits, and where it does not. Interkey, an official Datadog partner in Saudi Arabia, sells licences, implements and supports it.
Published 10 min read
The problem in plain words
- The usual failure: the dashboard loads, the numbers look normal, and they are yesterday's, or half of them are missing. Nobody is alerted, because nothing crashed.
- What data observability adds: checks on the data itself (freshness, row counts, blanks, schema) and on the jobs that move it, with lineage that shows which reports a problem reaches.
- Where Datadog fits: teams that run supported warehouses and pipeline tools and already use, or plan to use, Datadog for their applications and infrastructure.
- Where it does not: it does not decide what correct data is. Your data owners do, and the checks follow their rules.
Datadog Data Observability at a glance
| Question | Short answer |
|---|---|
| What it checks | Freshness, row counts, column metrics such as nulls and uniqueness, schema changes, and your own SQL checks |
| What it watches move | Jobs in Databricks, Spark, Airflow, dbt, AWS Glue and Azure Data Factory |
| Warehouses Datadog documents | Snowflake, Databricks, BigQuery, Redshift and ClickHouse Cloud, plus PostgreSQL |
| How problems are traced | Table- and column-level lineage, up to dashboards in tools such as Power BI, Tableau and Looker |
| What leaves your warehouse | The values of the checks, not table rows, according to Datadog's documentation |
| How Datadog bills it | Quality Monitoring per monitored table per month; Jobs Monitoring per host, per job or per DBU |
What data observability covers, and how it differs from data quality testing
Application monitoring asks whether a service is up and fast. A data pipeline can be up and fast and still deliver the wrong data, because most data failures are silent. An upstream system renames a column, a nightly extract times out half-way, a replication path falls behind. Every job reports success; the sales dashboard shows a quiet week.
Data observability watches five things, so that the data team hears about these problems before the people reading the report do:
Read more
- Freshness: did the table update when it should have?
- Volume: did the load bring roughly the usual number of rows?
- Quality of columns: nulls, duplicates, distinct values, ranges and averages that suddenly move.
- Schema: were columns added, removed, renamed or retyped?
- Lineage: what feeds this table, and which models and dashboards read from it?
Data quality testing is part of it, not all of it. A test checks a rule you thought of in advance, such as "customer ID is never empty", at the moment the pipeline runs. Monitoring watches for change you did not foresee, between runs, across tables nobody wrote tests for. Most teams need both.
Why it matters for organisations in Saudi Arabia
Three things make silent data failures expensive here. Management reporting increasingly runs on dashboards refreshed daily or hourly, so a stale table reaches decision-makers quickly. Data often reaches the warehouse through several hops, and each can fail without the others noticing. And Arabic text adds its own failure modes, such as names that arrive as question marks after an encoding change, which a check for empty values will not catch but a distinct-values or custom check can.
Read more
Residency also shapes the design. Datadog is a SaaS platform, and as of November 2026 its documented sites are outside the Middle East. Datadog documents that quality checks run as queries inside your warehouse and send only the resulting values, such as a row count or a count of empty values, not the rows. Two exceptions need a decision: a custom SQL check that groups by a column sends that column's values, and job logs can carry whatever a job prints. Saudi organisations weigh this against the Personal Data Protection Law, overseen by the Saudi Data and AI Authority (SDAIA), and their own classification; our Datadog data residency guide sets out the options.
RelatedSaudi Data and AI Authority (SDAIA)Datadog data residency guide
Data quality monitoring options and their trade-offs
Teams usually start with one of these and add the next when it stops being enough. None is wrong; each catches a different kind of problem.
| Option | What it catches | Trade-off |
|---|---|---|
| Tests inside the pipeline (for example dbt tests) | Rules you wrote in advance, at run time | Cheap and close to the code, but silent about tables without tests and about anything between runs |
| Scheduled SQL checks with email alerts | A few known risks on key tables | Quick to start; becomes a maintenance job as checks multiply, and has no lineage |
| Alerts from the orchestrator | Jobs that failed or ran late | Says the job broke, not whether the data it produced is right |
| A data observability platform | Freshness, volume, column and schema changes across many tables, jobs, and lineage to reports | Wider coverage and learned baselines, but a licence, a set-up project and alert tuning |
Whichever you choose, write down for each important report how fresh its data must be and who is told when it is not. Without that, every option produces alerts nobody owns.
How Datadog Data Observability does it
Datadog documents Data Observability in five parts. These are the jobs each one does, as Datadog describes them.
| Part | What Datadog documents |
|---|---|
| Quality Monitoring | Monitors for freshness, row count, column metrics (nullness, uniqueness, cardinality, minimum, maximum, mean and others) and custom SQL, on supported warehouses, data lakes and PostgreSQL |
| Schema change monitors | Alerts when columns or other structures are added, removed, renamed or retyped; listed by Datadog as in Preview |
| Jobs Monitoring | Failing jobs and their errors, run times per Spark job and stage, idle CPU and resource use, and alerts when a job fails or runs beyond its completion time |
| Lineage | Table- and column-level dependencies built from query history and job runs, with quality and job status on the graph |
| Data Catalog | A searchable inventory of data assets across the connected integrations |
| CI/CD | On a pull request that changes dbt models: the downstream assets affected, and drift in row counts, the share of empty values and cardinality before the change is merged |
Monitors can use static thresholds or anomaly detection. Datadog states that its anomaly model learns seasonality and trends, needs three to seven days of history to train, and suppresses alerts while it does. For lakes, Datadog documents AWS Glue and Amazon Athena; for Snowflake observability specifically, it documents support for Snowflake masking policies, so masked columns stay masked to the Datadog service account. We confirm what is available on your Datadog site during design.
Data pipeline monitoring when data arrives by replication
Many Saudi estates do not load the warehouse straight from the core system. A replication platform keeps a reporting copy close behind production, and the warehouse loads from that copy. Interkey's own iblync real-time replication works this way: it streams changes between systems that were not built to share data, with throttling and queueing.
The two answer different questions. iblync moves the changes and tells you about the replication itself. Data observability, on the target side, tells you whether the data the business reads is fresh and complete. Where the target is one Datadog supports, such as PostgreSQL or a supported warehouse, a freshness monitor on a replicated table shows lag in the terms that matter to a report: how old is the newest row? A row-count monitor shows a load that stopped half-way. Lineage then shows which dashboards read that table, so the alert reaches their owner.
Read more
The same split holds for event streams. Where data moves through Kafka or other queues, Datadog's Data Streams Monitoring measures end-to-end latency and consumer lag between services; our Kafka monitoring page covers that side. For the wider design of interfaces between systems, see our data integration services.
Relatediblync real-time replicationdata integration services
Where Datadog Data Observability fits, and where it does not
| Your situation | Fits? | Why |
|---|---|---|
| Snowflake, Databricks, BigQuery or Redshift, with dbt or Airflow | Yes | These are the platforms Datadog documents first, with lineage across them |
| Spark or Databricks jobs that are slow or costly | Yes | Jobs Monitoring shows stages, idle CPU and resource use next to infrastructure metrics |
| Applications and infrastructure already on Datadog | Yes | Data alerts, job runs and the hosts under them sit in one place, with one tagging standard |
| A replicated reporting copy on PostgreSQL feeding dashboards | Usually | Freshness and row-count monitors on the copy; check the PostgreSQL version against Datadog's list |
| A warehouse or lake Datadog does not list | Not yet | Datadog cannot query it; scheduled checks or job-level monitoring may cover part of the need |
| No agreed owner or freshness target for key reports | Not first | Monitors will fire with no one to act; agree owners and targets before buying anything |
What drives the cost
Datadog's list-pricing page states its units: Quality Monitoring per monitored table per month, and Jobs Monitoring per host per hour for clusters, per job per hour for orchestrators, and per DBU for Databricks serverless. Cost follows how many tables you monitor, not how many rows they hold, and how much compute your jobs use. Start with the tables behind the reports people act on, and widen once the alerts are trusted. List prices are on our Datadog pricing guide.
How Interkey helps: a data observability project, step by step
-
Pick the reports.
Three to five reports the business acts on, and for each one how fresh it must be and who owns it.
-
Trace what feeds them.
Sources, replication paths, jobs and tables, so the monitored set is the one that matters.
-
Quote the licences.
Datadog licences sized on monitored tables and job compute, with implementation and technical support in one offer.
-
Agree the data rules.
Which custom checks may group by which columns, what job logs may carry, and masking for the Datadog service account.
-
Connect and monitor.
Warehouse, pipeline and BI integrations, then freshness, volume, column and schema monitors, with thresholds where the rule is known and anomaly detection where it is not.
-
Route the alerts.
Each alert names the table, the reports affected and the first thing to check, and reaches the owner, not a shared inbox.
-
Operate or hand over.
Interkey operates the platform with support from Riyadh, in Arabic and English, or hands it over with documentation and stays on as technical support.
Questions buyers ask about data observability
Is data observability the same as data quality testing?
No. A data quality test checks a rule you wrote in advance when the pipeline runs. Data observability also watches for changes nobody foresaw, such as a table that stopped refreshing or a load with half the usual rows, and shows which reports they reach. Most teams keep their tests and add monitoring.
Which warehouses and pipeline tools can Datadog Data Observability monitor?
Datadog documents Snowflake, Databricks, BigQuery, Redshift and ClickHouse Cloud, AWS Glue and Amazon Athena for lakes, and PostgreSQL. For jobs it lists Databricks, Spark, Airflow, dbt, AWS Glue and Azure Data Factory. We check your versions against the current list before quoting.
Will our table data leave Saudi Arabia if Datadog monitors it?
Datadog documents that quality checks run inside your warehouse and send only the results, such as counts, not rows. A custom check grouped by a column sends that column's values, and job logs carry what jobs print, so we agree those rules with your data protection team first.
How is Datadog Data Observability licensed in KSA?
Datadog bills Quality Monitoring per monitored table per month and Jobs Monitoring by host, job or DBU. Interkey sells Datadog licences in Saudi Arabia and quotes them with implementation and technical support in one offer.
Can Datadog check the reporting copy that iblync replicates?
Yes, where the copy sits on a platform Datadog supports, such as PostgreSQL or a supported warehouse. Freshness and row-count monitors on the replicated tables show whether the copy the business reads is current and complete.
How long does it take to monitor the pipelines behind our main reports?
As an estimate, a few weeks for the tables behind a handful of reports on supported platforms. What drives it: how many sources feed them, agreeing owners and freshness targets, and the anomaly models' first days of training.
Next
Read next
Next step
Find the pipelines your reports depend on
Tell us which warehouse and pipeline tools you run, how data reaches the warehouse, and which reports matter most. We can quote the Datadog licences, the set-up and technical support together.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia