Datadog implementation guide · OpenTelemetry
Sending OpenTelemetry data to Datadog: choose the ingestion path, then govern the pipeline
Datadog accepts OpenTelemetry traces, metrics and logs through the Datadog Agent (with its embedded OpenTelemetry Collector, or by receiving OTLP directly), through an OpenTelemetry Collector you run yourself, or directly over OTLP where neither can run. The path you choose decides which Datadog features you get and where masking and sampling happen. This guide compares the paths against Datadog's current documentation and sets out the governed gateway pipeline Interkey uses for estates with services in several languages.
Published 11 min read
In short
- Datadog's recommended path: the Datadog Agent with the Datadog Distribution of OpenTelemetry (DDOT) Collector.
- Your own Collector: for teams that want to manage the pipeline themselves, including tail-based sampling and transformations. For new Collector configurations Datadog now recommends its OTLP HTTP exporter with the
span_metricsconnector; the older Datadog Exporter and Connector remain supported. - The Agent as an OTLP receiver: applications send OTLP to the Agent on port 4317 (gRPC) or 4318 (HTTP); log ingestion over OTLP is off by default.
- Application code does not change between paths: services instrumented with OpenTelemetry send the same OTLP data whichever path receives it.
The ingestion paths Datadog documents
| Path | What Datadog says it is for | Where processing runs | Check before choosing |
|---|---|---|---|
| Datadog Agent with DDOT Collector (Datadog's recommendation) | Teams that want OpenTelemetry neutrality together with Datadog capabilities such as Fleet Automation and Universal Service Monitoring | Inside the Agent on each host or node; on Kubernetes, Datadog also documents a DDOT gateway, a separate central Collector service for tail-based sampling and centralised processing | Availability for your Agent version and platform; the components included in DDOT, against the processors your rules need |
| Your own OpenTelemetry Collector | Teams that manage their own Collector; flexible configuration, including tail-based sampling and data transformations | In a Collector tier you operate, for example a gateway in Kubernetes | Needs the Collector Contrib distribution, v0.154.0 or later when we checked; not supported for serverless or task-based runtimes such as ECS Fargate, EKS Fargate and AWS Lambda |
| Datadog Agent receiving OTLP | Applications point at the Agent instead of a Collector | On each host or node, inside the Agent | Traces and metrics need Agent 7.32.0 or later, logs 7.48.0 or later; OTLP log ingestion is disabled by default "to prevent unexpected logs billing" and must be switched on deliberately |
| Direct OTLP ingestion | Environments where running an Agent or Collector is not feasible: serverless, managed platforms, tightly resource-constrained setups | In Datadog, apart from what each application's SDK does | No Collector or Agent in your network, so no central masking or tail-based sampling before data leaves your environment; only SDK-level controls, such as head sampling, apply |
Features depend on the path. Datadog states that the features available to you depend on these choices, and publishes a feature compatibility table. Read it against your must-have features, such as specific APM views or runtime metrics, before you standardise.
How we choose between them
Start with three questions:
- Does the Datadog Agent already run everywhere, and is Datadog the only destination? If yes, start with the Agent: DDOT if you want Collector pipelines managed with the Agent, or plain OTLP receiving for the simplest change.
- Do you need to send the same data to more than one destination, keep complete traces for errors and slow requests (tail-based sampling), or rewrite attributes centrally? If yes, you need a gateway tier: your own Collector, or on Kubernetes Datadog's DDOT gateway, provided its included components cover what you need, including exporters for any destination other than Datadog.
- Must personal data be masked inside your network for every language in the same way? Masking needs a processing tier you control inside your network, such as your own Collector gateway. If you plan to use DDOT instead, on each node or as a gateway, confirm it includes the processors your masking rules need. Direct ingestion alone cannot do it.
The paths can be combined: the Agent on each node for host and container data, and a Collector gateway for application telemetry from every language.
The reference pipeline for a multi-language estate
In Interkey's reference design, .NET 8, Java (Spring Boot) and Python (FastAPI and Flask) services send OTLP over HTTP to a Collector gateway of at least three replicas that scales with CPU load. The gateway sends traces, metrics and logs to your Datadog site over HTTPS.
-
OTLP/HTTP :4318
Python service .NET 8 service Java service -
Collector gateway (3+ replicas, autoscaled)
memory limiter resource detection health-check filter personal-data masking (tail sampling) batch -
Datadog site
HTTPS
Order matters: protect, enrich, filter, mask, sample, batch.
Inside the Collector, order matters:
| Stage | Why it is there |
|---|---|
| Memory limiter (always first) | Refuses data when the Collector nears its memory limit, instead of crashing |
| Resource detection | Adds host, cloud and container metadata |
| Health-check filter | Drops spans from liveness and readiness probes before they cost anything |
| Personal-data masking | Deletes, hashes or masks attributes and log text according to your policy |
| Tail sampling (optional) | Keeps complete traces based on outcome, such as errors and slow requests |
| Batch (last before export) | Groups data to reduce calls; Datadog's Exporter documentation requires a batch processor outside development environments and lists intake size limits, and batches that are too large are rejected with HTTP 413 |
Read more
Exporter choice in 2026. Interkey's reference design was built on Datadog's Exporter and Connector. For new Collector configurations Datadog now recommends its OTLP HTTP exporter with the span_metrics connector, and states that existing Exporter and Connector configurations remain supported and do not need to migrate. New gateways should follow Datadog's current recommendation; existing ones that work can stay as they are.
Accurate service metrics while sampling traces
Sampling keeps trace storage affordable, but request, error and latency metrics must reflect all traffic, not the sample. The fix is to split the work into two pipelines:
- Pipeline A receives every span and computes request, error and latency metrics from all traffic.
- Pipeline B applies sampling and sends only the kept traces to storage.
Datadog's own Collector example follows the same principle, separating trace processing from sampling "so span metrics are computed on all traces".
Read more
Tail-based sampling has a cost. Every span of a trace must reach the same Collector instance, so a first tier of load-balancing Collectors routes spans by trace ID to a second, sampling tier. The OpenTelemetry project notes that tail-sampling components must be stateful and can need significant compute.
A starting sampling policy by environment:
| Environment | Head sampling (in the application) | Tail sampling (in the Collector) | Reason |
|---|---|---|---|
| Development / test | 100% | Off | Full visibility while debugging |
| UAT / staging | 100% | Errors + slow + 25% baseline | Realistic performance testing |
| Production, normal volume | 100% | Errors + slow + 10–20% baseline | Every failure kept; predictable cost |
| Production, very high volume | 25–50% | Errors + slow + 5–10% of the remainder | Two-stage reduction for scale |
Logs and metrics are controlled separately: drop debug-level logs in the Collector, and use Datadog log indexes and exclusion filters to decide what is indexed and for how long. Excluded logs are removed from indexes but can still feed Live Tail, generate metrics and be archived. The billing side is covered in What drives your Datadog log bill.
Gateway details that cause production incidents
- The Datadog site must match your organisation. Set the site parameter to the site shown in the Datadog URL you log in to. A wrong site returns 403 even with a valid API key. Datadog sites are independent of each other, and data cannot be shared across sites.
- gRPC can pin traffic to one replica. gRPC keeps long-lived connections, so a plain Kubernetes ClusterIP Service may send one client's traffic to a single Collector replica. Prefer OTLP over HTTP from applications, or use a headless Service with client-side load balancing.
- Egress through a proxy. If the Collector accepts data but sends fail, outbound HTTPS to Datadog's intake for your site is usually blocked. Allow it, and set
HTTPS_PROXYon the Collector where egress is proxied. - Pin versions. Run a specific Collector image and Java agent release in production, not "latest".
- Keep the API key out of configuration files. Inject it from your secret manager or a Kubernetes Secret.
- One host-metrics source per node. Running host metrics in both the Collector and the Datadog Agent double-counts.
Verifying the pipeline, one hop at a time
Each check isolates one hop, so a failure points to its cause.
-
Collector is healthy:
the health endpoint answers and the logs show no "refused" or "denied" errors.
-
API key and site are valid:
Datadog's key-validation endpoint for your site returns valid.
-
The Collector accepts OTLP:
a single synthetic span posted to the Collector returns HTTP 200 and appears in Datadog APM under its test service name.
-
It holds under load:
a short load test with the OpenTelemetry telemetry generator produces traces, metrics and logs at a known rate.
-
Data flows through and out:
the Collector's own metrics show accepted and sent spans rising together (see the table below).
-
Pipelines are as configured:
the Collector's built-in debug pages list the expected pipelines and recent spans.
-
The application side is wired:
the service can reach the Collector, and its
OTEL_variables are actually set inside the container. -
Datadog returns the data:
a query from your CI pipeline finds the expected custom metric for the service and environment.
| Collector metric | Healthy signal |
|---|---|
| Accepted spans | Increasing |
| Refused spans | Zero; non-zero usually means the memory limiter is pushing back |
| Sent spans | Increasing, close to accepted, minus what was sampled or filtered |
| Failed sends | Zero; non-zero points to the key, site, network or proxy |
| Export queue size | Low and stable; constant growth means egress to Datadog is blocked or too slow |
Send the same Collector metrics to Datadog and put a monitor on failed sends, so the pipeline that watches your services is watched too.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Nothing in Datadog; Collector logs show 403 | Wrong API key or wrong site | Validate the key; set the site that matches your Datadog login URL |
| Collector accepts data but failed sends increase | Outbound HTTPS blocked by firewall or proxy | Allow egress to Datadog intake for your site; set HTTPS_PROXY on the Collector |
| Services appear without env or version | Resource attributes missing | Set OTEL_RESOURCE_ATTRIBUTES with the environment key(s) and service.version |
| Traces broken into separate pieces per service | Context propagation mismatch | Use the tracecontext and baggage propagators in every service |
| .NET manual spans missing | ActivitySource not registered | Register the source name in the tracing configuration |
| Logs not linked to traces | Logs sent outside OTLP without trace IDs | Export logs over OTLP, or add trace_id and span_id to the log format |
| Request counts in APM look too low | Metrics computed after sampling | Compute metrics from all spans before sampling |
| Refused spans increasing | Collector memory pressure | Add replicas, raise the memory limit or reduce batch size |
| Duplicate host metrics | Collector and Agent both collecting host metrics | Choose one host-metrics source per node |
What an OpenTelemetry-to-Datadog engagement covers
Interkey is an official Datadog partner and implements and operates Datadog from Riyadh. On a pipeline engagement we work through this checklist with your teams, and each item has a named owner on your side:
| # | Task | Typical owner |
|---|---|---|
| 1 | Approve the tagging standard (service, env, version, team) | Architecture |
| 2 | Store the Datadog API key in the secret manager; confirm the Datadog site | Platform |
| 3 | Deploy the ingestion path chosen above (Agent with DDOT, Collector gateway, or both) | Platform |
| 4 | Pass verification checks 1–6 | Platform |
| 5 | Pilot one Python service | Application team |
| 6 | Pilot one .NET service | Application team |
| 7 | Pilot one Java service | Application team |
| 8 | Prove one cross-language distributed trace end to end | Architecture |
| 9 | Review personal-data masking rules with security and the data protection officer | Security |
| 10 | Agree the sampling policy per environment; measure ingest | Platform and finance |
| 11 | Build the pipeline-health dashboard and monitors | SRE |
| 12 | Add business spans and metrics for the top three customer journeys | Application teams |
New to OpenTelemetry itself? Start with OpenTelemetry: what it standardises and how to adopt it.
Questions buyers ask
Do we need the Datadog Agent if we run our own Collector?
No. Datadog documents that a Collector can send traces, logs and metrics to Datadog without the Agent. Some Datadog capabilities are tied to the Agent and DDOT, so check the feature compatibility table for what you need.
We already use the Datadog Exporter. Do we have to migrate?
Datadog states that the Datadog Exporter and Connector remain supported and that existing configurations do not need to migrate. For new Collector configurations it recommends the OTLP HTTP exporter with the span_metrics connector.
Will switching on OTLP logs increase our bill?
It can. That is why log ingestion over OTLP is disabled by default on the Agent. Decide which logs are indexed, and for how long, before you enable it: What drives your Datadog log bill.
Can the same telemetry go to Datadog and to another destination?
Yes, if you run your own Collector: one pipeline can export to several destinations, such as Datadog and an internal data lake, without changing application code.
Is there a Datadog site in Saudi Arabia?
Not as of November 2026. Datadog's sites are in the United States, Germany, Japan, Australia and the United Kingdom. Masking in your own network before export is covered in Keeping personal data out of logs and traces, and the wider residency question in Observability data residency in Saudi Arabia.
RelatedKeeping personal data out of logs and tracesObservability data residency in Saudi Arabia
Next
Related guides
Next step
Review your OpenTelemetry-to-Datadog design
Tell us which ingestion path you use or are considering, the languages and platforms involved, and what is not working. A specialist from the Riyadh team replies on Saudi working days, in Arabic or English.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia