Datadog · Observability Pipelines
Observability pipelines in Saudi Arabia: filter, mask and route logs before they leave
Observability pipelines process logs and metrics inside your own network before they are sent anywhere. With Datadog Observability Pipelines, a Worker that you run in your data centre or cloud account filters noise, masks personal data and routes each log to Datadog, an archive or your security tools. For a Saudi organisation that means two decisions are taken before data leaves the Kingdom: what is worth paying to index, and what must never leave at all. Interkey is an official Datadog partner in Saudi Arabia: we sell the licences, design and deploy the pipelines with your teams, and support them after go-live.
Orientation
At a glance
The page in one table, for the reader who has five minutes.
- What it is
- Datadog Observability Pipelines: a Worker in your own infrastructure that collects, processes and routes logs and metrics
- Where it runs
- Your data centre or cloud account: Docker, Kubernetes, Linux servers, Amazon EC2 or Amazon ECS Fargate
- What it does to logs
- Filters, samples, deduplicates, caps daily volume, parses, enriches, masks sensitive data and turns logs into metrics
- Where logs can go
- Datadog, a Datadog archive in your own storage bucket, your SIEM, Kafka, syslog or an HTTP endpoint, or several at once
- Why Saudi teams use it
- To keep personal data and chosen logs from leaving the Kingdom, and to stop paying to index logs nobody reads
- What it does not cover
- Traces: Datadog documents it for logs and metrics
- What Interkey does
- Licences, volume review, pipeline design, Worker deployment, masking rules, monitoring of the pipelines and technical support from Riyadh
Observability pipelines: deciding before the data leaves
In most estates every server and application sends every log line to the monitoring platform, and decisions come afterwards: what to index, what to drop, which line held a national ID number. By then the data has left your network.
A telemetry pipeline moves those decisions earlier. Datadog describes Observability Pipelines as a way to "collect and process logs and metrics within your own infrastructure": the Worker receives logs from your existing agents and shippers, applies the rules you define, and only then sends each log where it belongs. In Datadog, the Pipelines screen builds pipelines from templates and manages the Workers.
If you run Datadog Log Management, the pipeline sits in front of it and decides what reaches it.
Log pipeline templates and what they solve
Datadog ships each common job as a template. Most pipelines combine two or three.
| Template | What it does | Typical reason |
|---|---|---|
| Log Volume Control | Filters out low-value logs before they are stored and indexed | The log bill grows faster than the estate |
| Sensitive Data Redaction | Finds and removes personal data and secrets with built-in or custom rules | Personal data must not leave the network |
| Archive Logs | Writes raw logs to object storage for long retention and later rehydration | Audit and security logs kept for long periods at low cost |
| Dual Ship Logs | Sends one log stream to more than one destination | Datadog for operations and a SIEM for the security team |
| Split Logs | Routes logs by type to different tools | Security logs to one place, application logs to another |
| Generate Log-based Metrics | Turns high-volume logs into count or distribution metrics | The trend matters, not every line |
| Metric Tag Governance | Keeps needed metrics, standardises tags and removes unwanted ones | Custom metric cardinality out of control |
Each pipeline is a source, an ordered list of processors and one or more destinations, and every part can be changed.
Log routing and how to reduce log volume
Cut volume with processors, not guesses
Datadog's documented log processors include Filter, Sample, Deduplicate, Throttle, Quota and Reduce. Filter drops what matches a query, such as health checks or debug lines; Sample keeps a share of a noisy stream.
The Quota processor sets a daily limit on matching logs, by number of events or by bytes, and can be split by service, environment or status. When a limit is reached, the pipeline can drop the extra logs, keep them, or send them to overflow storage in Amazon S3, Azure Blob or Google Cloud Storage. Datadog notes that counts between Workers sync a few times a minute, so a burst can pass the limit, and recommends pairing Quota with Throttle.
Read more
Route each log class deliberately
Log routing is where cost and retention meet. Errors and the request logs of critical services go to Datadog for search and alerting. Audit and security logs go to an archive in storage you own, and are rehydrated when an investigation needs them. Our Datadog log cost guide explains how to assign each class without deleting the audit trail.
We do not quote a saving before measuring your logs.
RelatedDatadog log cost guide
Masking personal data in the telemetry pipeline
The Sensitive Data Scanner processor runs inside the Worker, so matching values are changed before the log leaves your premises. It uses Datadog's library rules or your own regular expressions. On a match it can redact the value with replacement text, partially redact it, or hash it into a unique identifier. Datadog states that all three actions are irreversible, which is the point: what is masked in the Kingdom cannot be recovered elsewhere.
Local patterns, such as Saudi national ID and Iqama numbers, mobile numbers and IBANs, are written as custom rules and tested against real samples before they are switched on. Our telemetry PII redaction guide sets out how this layer fits with masking at source and in Datadog.
Data residency: what stays in the Kingdom and what leaves
Datadog is SaaS, and as of November 2026 its documented sites are in the US, Germany, Japan, Australia and the UK, none in the Middle East. Observability Pipelines does not change where Datadog stores what you send it. What it changes is what you send.
- It needs an outbound connection to your Datadog site. Datadog's installation guide requires an API key enabled for Remote Configuration and lists the Datadog endpoints a firewall must allow.
- Logs that must stay go somewhere else. Archives, overflow storage and other destinations are ones you select, so logs that must not leave can be routed to storage or tools inside the Kingdom.
Read more
The Personal Data Protection Law is overseen by the Saudi Data and AI Authority (SDAIA), and the National Cybersecurity Authority (NCA) sets controls that cover log retention and monitoring. Their own texts are the reference. Our Datadog data residency guide covers the wider options, and the data residency in Saudi Arabia guide covers the principle beyond monitoring.
Interkey's engineers work from Riyadh, in Arabic and English, and write the routing and masking rules with your security and compliance owners.
RelatedDatadog data residency guidedata residency in Saudi Arabia guide
Where a pipeline fits, and where it does not
- It fits when log volumes are large, several teams need the same logs in different tools, or residency rules decide what may leave.
- It may not be needed when volumes are modest and Datadog is the only destination: exclusion filters and masking in the Datadog Agent can be enough.
- It does not fix bad logging. An application that logs every request at debug level is corrected at source.
- It does not process traces. For traces, the OpenTelemetry Collector does the in-network work.
Sources include the Datadog Agent, Fluentd and Fluent Bit, syslog, HTTP, Kafka and OpenTelemetry, so existing shippers usually stay.
How Interkey implements Observability Pipelines
-
Measure the logs.
Sources, daily volume per service, and what nobody searches.
-
Agree the classes.
For each class of log: indexed, archived, a metric, kept in the Kingdom or dropped.
-
Write the masking rules.
Library rules plus custom rules for local identifiers, tested on real samples.
-
Quote the licences.
Observability Pipelines with the other Datadog products, set-up and support, in one offer, after measuring volumes. Our Datadog pricing guide explains the wider bill.
-
Deploy the Workers.
Datadog documents Docker, Kubernetes through its Helm chart (including Amazon EKS, Azure AKS, Google GKE, Red Hat OpenShift and Rancher), Linux APT or RPM packages, Amazon EC2 through CloudFormation, and Amazon ECS Fargate. Sized for peak volume, with the firewall rules agreed.
-
Move sources over in stages.
One service at a time, comparing volumes.
-
Operate or hand over.
Datadog recommends updating the Worker at least monthly, so support includes an update routine. Interkey supports the pipelines from Riyadh, or hands them over with documentation.
Questions buyers ask
Can we mask and filter logs in Saudi Arabia before they reach Datadog?
Yes. The Observability Pipelines Worker runs in your own data centre or cloud account, so filtering and masking happen there. The Sensitive Data Scanner processor redacts, partially redacts or hashes matching values before the log is sent on.
Does an observability pipeline keep our log data inside the Kingdom?
Only the logs you choose not to send. What reaches Datadog is stored on your Datadog site, and as of November 2026 none is in the Middle East. Logs routed to an archive, overflow storage or a tool inside the Kingdom stay where you send them. The Worker itself needs an outbound connection to Datadog for its configuration.
How much log volume can a log pipeline remove?
It depends on how noisy your sources are and what your security and compliance owners must keep. We measure daily volumes per service first and agree each class of log, then report the volume before and after each source moves.
How are Observability Pipelines licences bought in Saudi Arabia?
Through Interkey, as an official Datadog partner in Saudi Arabia. We quote Observability Pipelines with your other Datadog products, the set-up and technical support in one offer, after measuring your log volumes.
Can one pipeline send the same logs to Datadog and to our SIEM?
Yes. The Dual Ship Logs template sends one stream to several destinations, and Split Logs sends security logs and application logs to different tools.
How long does it take to deploy Observability Pipelines?
It depends on the number of log sources, how quickly the log classes and masking rules are agreed, and the change process on your platform. We give an estimate after the volume review and move sources over in stages, starting with the noisiest.
Next
Read next
Next step
Design your log pipeline
Tell us roughly how many logs you send each day, which shippers you use, and what must stay in the Kingdom. We can quote the Datadog licences, the pipeline design and technical support together.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia