Datadog · Log Management
Datadog Log Management, implemented: collection, parsing, indexes and archives
Datadog Log Management collects logs from your hosts, containers, cloud services and applications, parses them into searchable fields, and lets you search, alert on and archive them in one place. It separates ingesting a log from indexing it, so you can collect everything and still decide, log by log, what stays searchable, what is kept long term and what only feeds a metric. Interkey, an official Datadog partner, sells Datadog licences in Saudi Arabia, implements Log Management and provides technical support after go-live.
Published 7 min read
In short
- What it does: collects, parses, searches, alerts on and archives logs, alongside Datadog metrics and traces.
- How logs arrive: the Datadog Agent on hosts and containers, cloud and serverless integrations, log shippers, or the HTTP API.
- How they become useful: pipelines and processors turn raw lines into named fields you can filter and group by.
- Where they go: a standard index for daily troubleshooting, Flex Logs for longer, occasional search, an archive in your own storage, or a metric only.
- Personal data: mask it before it leaves your network, or scan and redact it in Datadog.
- What Interkey does: licences, collection, parsing standards, routing and retention per log class, access rules, alerts, and support from Riyadh.
How Datadog Log Management works
| Stage | What happens | What you decide |
|---|---|---|
| Collect | The Agent, an integration or the API sends each log to your Datadog site | Which sources, and which logs are filtered or masked before sending |
| Process | Pipelines parse each log and map its date, host, service, status and message | The attribute names every team will use |
| Route | Each log is sent to an index, to Flex Logs, to an archive, into a metric, or to several of these | Retention and searchability per class of log |
| Explore and alert | Search, Live Tail, patterns and analytics; monitors raise alerts | Who sees which logs, and which conditions page someone |
Logs link to traces and metrics, so a slow request in Datadog APM leads straight to the log lines behind it.
Getting logs into Datadog
| Source | How logs are collected |
|---|---|
| Linux and Windows servers | The Datadog Agent with log collection turned on. It tails files, listens on TCP or UDP, masks values and joins multi-line entries such as stack traces |
| Containers | The Agent reads container output on Docker, Kubernetes, Red Hat OpenShift, Amazon ECS and ECS Fargate, configured per container through labels |
| Applications | Your existing logger, with the Agent collecting its output. Datadog documents Java, Python, Go, Ruby, .NET, PHP, Node.js and mobile platforms |
| Cloud services and serverless | Cloud integrations, and a Lambda-based forwarder for serverless workloads |
| Existing shippers and anything else | Syslog and other common shippers, or the Datadog HTTP log API over HTTPS |
Three limits matter. The API accepts logs of up to 1 MB, and Datadog recommends 25 KB or less per log. Logs with a timestamp more than 18 hours old are rejected. And log ingestion over OpenTelemetry (OTLP) through the Agent is off by default.
Parsing logs with pipelines
Datadog pipelines run processors in order on every log that matches the pipeline's filter. A Grok parser extracts fields from free text, and remappers set the official date, service, status, message and trace ID.
- Integration pipelines are added automatically for known sources. They are read-only; to change one, clone it and edit the clone.
- Filters run before processors, so a pipeline cannot select logs by a field it extracts itself.
- Keep pipelines small. Datadog recommends no more than 20 processors per pipeline and 10 parsing rules per Grok parser.
- Agree one naming standard, so
status,serviceandenvmean the same thing in every team's logs.
Where each log should go
Because ingestion and indexing are separate, every ingested log can appear in Live Tail, feed metrics and be archived, indexed or not. Most of the design work sits here.
| Destination | Use it for | Retention | Searching and alerting |
|---|---|---|---|
| Standard index | Application and service logs used for daily troubleshooting | Set per index | Full search, and log monitors |
| Flex Logs | Security, transaction and network logs kept for months and searched now and then | 30 to 450 days with scalable compute; 3, 6, 12 or 15 months with Flex Logs Starter | Searchable; monitors and Watchdog are not supported |
| Archive | Audit and configuration logs kept for a long time and rarely read | As long as your storage keeps them | Rehydrate or search the archive when needed |
| Log-based metric | High-volume logs where the trend matters more than each line | Metrics kept at 10-second granularity for 15 months | Graphs and metric alerts |
- Indexes each have their own retention, and exclusion filters keep chosen logs out of an index without dropping them from Live Tail, metrics or archives.
- Archives receive all ingested logs, indexed or not, and are written to a storage bucket that you own in Amazon S3, Google Cloud Storage or Azure Storage, as compressed JSON. Each archive can have a filter.
- Log-based metrics are calculated from the whole ingest stream and billed as custom metrics, so avoid grouping them by user IDs or request IDs.
Our Datadog log cost guide explains how these choices drive the bill, and how to set retention per class of log.
Searching logs and alerting on them
In the Log Explorer, teams search, group and graph logs, view patterns, and save the views they use often. Live Tail shows indexed and non-indexed logs in near real time.
A log monitor alerts when matching logs cross a threshold in a time window: a log count, a count of unique values, or a numeric value such as response time. Two rules shape alert design:
- Log monitors evaluate indexed logs only, and only the standard tier. Logs kept only in Flex Logs or an archive cannot trigger them.
- The longest rolling window is 2 days. For longer trends, alert on a log-based metric instead.
Personal data and access in logs
Logs often carry names, phone numbers, national ID numbers and card numbers nobody meant to collect. Deal with them in three places:
- In the code. Don't log request bodies or personal fields at all.
- Inside your network. Observability Pipelines runs a Worker in your own infrastructure that can redact personal data and secrets before logs leave.
- In Datadog. Sensitive Data Scanner finds and redacts, partially redacts or hashes sensitive values. In the cloud it scans after submission, so the data leaves your environment before it is redacted.
Read more
Access can also be limited by role, with restriction queries that let a team see only the logs that match its query. As of November 2026, Datadog lists no data-centre site in the Middle East. Our Datadog data residency guide covers what reaches Datadog and where it is stored. Interkey also offers a telemetry PII masking review: we test your existing masking rules against Saudi formats, such as mobile and national ID numbers, and against values that only look similar.
RelatedDatadog data residency guide
How Datadog bills for logs
Datadog bills standard Log Management on two units: gigabytes ingested, and millions of log events indexed. Flex Logs storage is billed per million events stored and its compute at a flat rate per compute size; Flex Logs Starter is billed at a bundled storage and compute rate. Log-based metrics count as custom metrics. Your Datadog quote sets the price.
How Interkey implements Log Management
- Log inventory. Sources, daily volume, users, and classes such as application, security, audit and debug.
- Licences. A Datadog licence quote for the volumes and retention you need.
- Collection. Agent and integrations, masking at source where needed, multi-line handling.
- Parsing standard. Pipelines, a shared attribute naming standard, and logs linked to traces.
- Routing and retention. Indexes, exclusion filters, Flex Logs, archives and log-based metrics per class.
- Access and alerts. Restriction queries, masking rules, and the monitors that should page someone.
- Support. Technical support after go-live, regular reviews of volume and noisy alerts, and operation from Riyadh or a documented hand-over.
Interkey assesses, implements and operates Datadog for organisations in Saudi Arabia. The implementation team is in Riyadh and works in Arabic and English. Read about Interkey's Datadog implementation.
Questions teams ask
Do we have to index every log we send?
No. Ingestion and indexing are separate. Logs you exclude from indexes still pass through Live Tail, can generate metrics and are still archived.
Can logs stay inside the Kingdom?
Logs you send are processed on your Datadog site, and as of November 2026 none is in the Middle East. To keep certain fields or logs from leaving at all, mask or drop them inside your network, at source or in Observability Pipelines.
Does Interkey sell Datadog licences?
Yes. Interkey sells Datadog licences, implements Datadog and provides technical support in Saudi Arabia.
We already send logs to Datadog. Can you help?
Yes. We start by reviewing what you collect, how it is parsed and routed, and which alerts fire, then fix the gaps.
Next
Related pages
Next step
Request a log management plan
Tell us which systems produce your logs, roughly how much they produce each day, and how long each kind must be kept. We will reply with a proposed collection and routing plan and the licences it needs.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia