Datadog · Distributed tracing
Distributed tracing: following one request across your microservices, and deciding what to keep
Distributed tracing follows a single request as it passes through every service, queue, database and external API that handles it, and shows where the time went and where an error started. In a microservices estate it is often the only way to answer "which service made this slow". This guide explains how tracing works, why sampling decides both cost and usefulness, how OpenTelemetry and the Datadog Agent compare, and what it means for teams in Saudi Arabia.
Published 9 min read
Orientation
At a glance
The whole page in one table, before the detail.
- What it is
- A record of one request across services, made of timed spans sharing a trace ID
- What it answers
- Where the time went and where the error started
- What makes it work
- Every service passes the trace context on in its outgoing calls and messages
- What drives the cost
- How many spans you send and how many you index and keep
- How it is collected
- Datadog SDKs or Single Step Instrumentation, or OpenTelemetry, into Datadog APM
- Saudi angle
- Spans can carry personal data, and as of November 2026 no Datadog site is in the Middle East
The problem in plain words
- The usual situation: a customer journey is slow, every team's dashboard looks normal, and nobody knows which service is responsible.
- What tracing adds: one record per request, with a timed step for every service it touched, joined by a shared trace ID.
- The real decision: not whether to trace, but which traces to keep. Keeping everything is expensive; keeping too little loses the errors you need.
- Standards: OpenTelemetry and the Datadog Agent both collect traces for Datadog; the choice affects masking and sampling, not your business logic.
What distributed tracing is: one request across microservices
When a request arrives, the first service that sees it starts a trace and gives it an ID. Each unit of work along the way, an HTTP handler, a database query, a call to a payment provider, a message put on a queue, becomes a span: a named step with a start time, a duration, a status and some attributes. Because each span records its parent, the set can be drawn as a flame graph showing what ran in sequence and what ran in parallel.
Read more
The part that is easy to get wrong is context propagation. A trace only stays whole if every service reads the trace context from the incoming request and writes it into every outgoing call. Over HTTP that is a header; the W3C Trace Context standard uses one called traceparent. Over a queue it has to travel in the message headers. Datadog documents that its SDKs, by default, read and write Datadog's own headers, W3C Trace Context and baggage. When one service uses a different format, or a queue client drops the headers, the trace breaks into pieces.
Tracing does not replace metrics or logs. Metrics say error rates rose; a trace says which request failed and where; the logs of that request say what the code said.
Microservices tracing: what one trace records
A checkout request in a typical estate, as a single trace would show it. Every box is a span; the trace ID ties them together.
-
Mobile app or browser
the request starts; a trace ID is created or passed on
-
API gateway
first server-side span
-
Services the request calls
orders service pricing service payment service -
Dependencies
database query external payment API message to a queue -
Datadog APM
one trace, every span, with timings and errors
The trace context travels with the request, through HTTP calls and queue messages alike.
Why tracing matters for teams in Saudi Arabia
Banks, government entities and retailers in the Kingdom run customer journeys through apps that call many services: identity checks, payment gateways, core systems. When one step slows, the customer sees one slow app, and every team reports that its own part is fine.
Read more
Two local questions come up early. First, data residency: span attributes can contain URLs, SQL statements and error messages that hold a national ID, an Iqama or a mobile number. Datadog is a SaaS platform, and as of November 2026 none of its sites is in the Middle East, so personal data has to be kept out of spans or masked inside your network before it is sent. Our Datadog data residency guide covers the options. Second, ownership: someone has to own the tagging standard and the sampling policy for the whole estate. Interkey's engineers in Riyadh work through both with you.
RelatedDatadog data residency guide
OpenTelemetry tracing vs the Datadog Agent: the options and their trade-offs
Datadog accepts traces from its own SDKs and from OpenTelemetry. The path decides where processing happens.
| Option | How it works | Good for | Watch out for |
|---|---|---|---|
| Datadog SDKs, or Single Step Instrumentation | Datadog tracing libraries send spans to the Agent on each host or node | Estates where Datadog is the only destination and the Agent already runs | Check the compatibility list for each language and framework you run |
| OpenTelemetry SDKs into the Datadog Agent with DDOT | The Agent, with its embedded Datadog Distribution of OpenTelemetry Collector, receives OpenTelemetry data | Vendor-neutral instrumentation on Datadog's recommended path | Available features depend on the set-up; read Datadog's compatibility table |
| OpenTelemetry SDKs into your own Collector gateway | A Collector tier you run masks, samples and exports traces | Central masking, tail-based sampling, or more than one destination | You operate and scale the Collector |
| Direct OTLP ingestion | Applications send OpenTelemetry data straight to Datadog | Serverless platforms where nothing else can run | No central masking or tail sampling inside your network |
The paths can be combined, and application code does not change between them. We compare the paths against Datadog's documentation in sending OpenTelemetry data to Datadog, and explain the standard itself in OpenTelemetry: what it standardises and how to adopt it.
Trace sampling: keeping the traces that matter at a cost you can predict
A busy service can produce millions of near-identical spans an hour. Sampling decides which traces are kept, in one of two ways.
- Head-based sampling decides at the start of the request. Datadog documents that the keep-or-drop decision is made at the start of the root span and passed to the other services through the request context, so a trace is kept or dropped whole. By Datadog's own default, the Agent adjusts rates toward a target of ten traces per second overall, and a separate error sampler catches up to ten traces per second per Agent that contain errors.
- Tail-based sampling decides after the trace is complete, so it can keep every trace that ended in an error or ran slow. It needs every span of a trace to reach the same Collector, so a load-balancing tier routes spans by trace ID.
Sampling must not distort service metrics. Datadog calculates request, error and latency metrics from all traces; in your own Collector, compute them from every span before the sampling step.
A starting sampling policy by environment
This is the policy Interkey starts from in its OpenTelemetry reference design. It is tuned per service once real volumes are measured.
| Environment | Head sampling (in the application) | Tail sampling (in the Collector) | Reason |
|---|---|---|---|
| Development and test | 100% | Off | Full visibility while debugging |
| UAT and staging | 100% | Errors, slow traces and a 25% baseline | Realistic performance testing |
| Production, normal volume | 100% | Errors, slow traces and a 10–20% baseline | Every failure kept, at a predictable cost |
| Production, very high volume | 25–50% | Errors, slow traces and 5–10% of the rest | Two-stage reduction for scale |
In Datadog, what you send and what you keep are separate. All ingested spans can be searched live for 15 minutes; spans chosen by your retention filters stay searchable for 15 days; and Datadog's intelligent retention filter keeps a diverse sample, biased to errors and high latency, for 30 days. Our Datadog APM page sets out the retention table in full.
What drives the cost of distributed tracing
Datadog bills APM per APM host, per month, and each APM host includes an allowance of 150 GB of ingested spans and 1 million indexed spans a month. The bill therefore follows three things you control: how many hosts run traced services, how many spans those services send, and how many spans your retention filters index. Health-check spans are the easiest to drop. Prices are on our Datadog pricing guide; we quote for your estate.
Read more
As an estimate, a first traced journey across a few services takes weeks; a whole estate takes longer. What drives it: the number of languages, how many services use queues, whether the Agent or a Collector already runs, and how fast owners add missing trace context.
Distributed tracing tools: a checklist before you choose
Questions to put to any tracing set-up, including ours.
| Question | Why it matters |
|---|---|
| Does every language and framework you run have supported instrumentation? | One untraced service breaks every trace through it |
| Which propagation formats does it read and write? | Mixed formats across teams split traces into pieces |
| Does trace context survive your queues and async jobs? | Traces break most often at message-driven steps |
| Where does sampling happen, and can it keep every error? | Head sampling alone can drop the failing request |
| Are service metrics computed before sampling? | Otherwise request and error counts are understated |
| Where can personal data be masked before it leaves your network? | Spans carry URLs, SQL and error messages |
| Do traces link to logs, user sessions and database queries? | A trace shows where; linked data shows why |
| What does the bill depend on, and who watches it? | Span volume grows with traffic and services |
Where Datadog APM fits, and where it does not
Datadog APM fits when you want traces beside your metrics, logs, real user monitoring and database monitoring, linked by the same env, service and version tags. From a trace an engineer opens that request's logs, the user session behind it, the query it waited on and, with Continuous Profiler, the methods that used the CPU.
Read more
Where it may not fit: if personal data must never leave the Kingdom in any form and cannot be masked reliably, a SaaS platform with no Middle East site is a hard constraint, and that conversation comes before any tool choice. For a single monolith, good logs and database monitoring may do more. If you need the same traces in several destinations, plan an OpenTelemetry Collector gateway rather than relying on one vendor's agent alone. We will tell you which applies to you.
How Interkey runs a tracing project
-
Pick one journey.
One the business notices within minutes when it slows.
-
Agree the standard.
Service names, environments, versions and one propagation format for every team.
-
Choose the path.
Datadog SDKs, the Agent with DDOT, a Collector gateway, or a mix, per service.
-
Mask personal data.
Rules tested against Saudi formats before production is traced.
-
Set sampling and retention.
A policy per environment, sized against the span allowances, with every error kept.
-
Prove one trace end to end.
Across languages and queues, linked to its logs.
-
Operate or hand over.
Interkey, an official Datadog partner in Saudi Arabia, sells the licences, implements and supports them from Riyadh, or hands over with documentation.
Questions buyers ask about distributed tracing
What is the difference between distributed tracing and APM?
Distributed tracing records one request across services. APM is the wider product built on it: service metrics, deployment comparisons, profiling and links to logs.
Why do our traces break into separate pieces between services?
Usually because the trace context is not passed on. One service uses a different propagation format, or a queue client drops the message headers. Use one format across teams and check every asynchronous hop.
Should we instrument with OpenTelemetry or the Datadog SDKs?
Both send traces to Datadog. The Datadog SDKs are simplest where the Agent already runs; OpenTelemetry stays vendor-neutral and allows a Collector gateway for masking and tail sampling.
Will sampling make us miss the errors we need to investigate?
It should not, if it is designed for that. Keep every error and slow trace with an error sampler or tail-based sampling, sample the healthy baseline, and compute service metrics before sampling so counts stay accurate.
Can trace data stay inside Saudi Arabia?
Traces sent to Datadog are stored on your Datadog site, and as of November 2026 none is in the Middle East. Keep personal data out of span attributes, or mask it in a Collector inside your network before it is sent.
How long does it take to trace our microservices end to end, and what does it cost?
A first journey across a few services takes weeks; a whole estate takes longer, depending on languages, queues and existing agents. Cost follows APM hosts and span volume. Interkey quotes licences, set-up and support from Riyadh together.
Next
Read next
Next step
Plan distributed tracing for your services
Tell us the journey you want to see end to end, the languages behind it, and whether Datadog or OpenTelemetry already runs. We reply with a tracing path, a sampling policy and a first scope.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia