Datadog · LLM Observability
LLM observability with Datadog: prompts, latency, token cost and quality
LLM observability shows what a generative AI application actually did on each request: the prompt it built, the documents it retrieved, the tools an agent called, how long each step took, how many tokens it used, what it cost, and whether the answer was any good. Datadog LLM Observability records this as traces you can search, measure and evaluate. Interkey, an official Datadog partner, sells Datadog licences in Saudi Arabia, instruments your GenAI applications, and provides technical support after go-live.
In short
- The problem: a GenAI application can be up, fast and still wrong, and server monitoring will not show it.
- What Datadog records: each request as a trace, with a span for every model call, retrieval, tool call and agent step, plus latency, errors, token usage and cost.
- How quality is checked: managed evaluators, your own LLM-as-a-judge evaluators, results from your own tests, and end-user feedback.
- How it pairs with Kong AI Gateway: the gateway controls and records every call to a model; LLM observability explains what happened inside the application around that call.
- What Interkey does: licences, instrumentation, evaluation design, personal-data controls, dashboards, alerts and technical support from Riyadh.
Datadog LLM Observability at a glance
| Question | Short answer |
|---|---|
| What is traced | A single model call, a fixed workflow, or an agent run, with a span for each step |
| What is measured | Latency, errors, token usage and cost per request and per step |
| How applications are instrumented | SDKs for Python, Node.js and Java, or Datadog's HTTP API for other languages |
| How quality is judged | Managed and custom evaluators, external evaluation results and user feedback |
| How it is billed | On ingested LLM spans, according to Datadog's documentation |
| Where the data goes | To your Datadog site; as of November 2026 Datadog lists no site in the Middle East |
What LLM observability must cover
Ordinary application monitoring asks whether a service is up and how fast it responds. A generative AI application fails more quietly: the model answers in the wrong language, retrieval finds an out-of-date policy, or an agent calls the same tool five times, and every request still returns in two seconds.
LLM monitoring, AI observability and GenAI monitoring are names buyers use for the same need. In practice it covers six things:
Read more
- The full path of a request: the prompt that was built, the context that was retrieved, each model call and each tool call, in order.
- Latency per step, so you can see whether the time went to the model, the vector search or your own API.
- Tokens and cost, per request, per feature and per model, so finance can see where AI spend comes from.
- Errors, including provider timeouts, refused requests and failed tool calls.
- Answer quality, judged by rules, by a model acting as a judge, or by people.
- Safety and privacy, such as prompt-injection attempts and personal data in prompts or answers.
Teams usually start with one of four applications:
- An internal assistant over HR or policy documents: retrieval quality, cost per department, and answers in the right language.
- A customer-facing Arabic chatbot: slow or failed answers, personal data in transcripts, and user feedback.
- Search over company documents (RAG): whether the retrieved context was used, and which questions have no good source.
- Agents that call internal APIs: loops, repeated tool calls and steps that drive cost.
ML observability, monitoring models you train yourself for drift and accuracy, is a related but different discipline.
How Datadog LLM Observability does it
Datadog's documentation currently titles the product Agent Observability; its links and many references still use the name LLM Observability. These are the jobs it documents.
| Job | What Datadog documents |
|---|---|
| Trace | Each request becomes a trace; spans cover each agent decision or workflow step |
| Capture | Auto-instrumentation records input and output messages, token usage when available, latency, errors and model parameters |
| Measure | Out-of-the-box dashboards for cost, latency, performance and usage trends |
| Evaluate | Managed evaluators such as Language Mismatch and Sensitive Data Scanning, custom LLM-as-a-judge evaluators, and external evaluations sent through the API |
| Review | End-user feedback and annotation queues for structured human review of traces |
| Understand traffic | Patterns groups production questions into topics, to show what users ask and how answer quality changes over time |
| Find recurring problems | Insights analyses traces for repeated cost and reliability issues, each with a root cause, evidence and a suggested fix |
| Protect data | Sensitive Data Scanner checks inputs and outputs and redacts sensitive information it finds; prompt-injection attempts are identified |
Feature names change between releases; we confirm what is available on your Datadog site during design.
Instrumenting a GenAI application
Datadog provides SDKs for Python (3.7 and later), Node.js (16 and later) and Java (8 and later). Each supports auto-instrumentation, which captures model calls in supported frameworks without code changes, and custom instrumentation, which lets your developers mark workflows, agent steps, tool calls and sessions themselves. Applications in other languages can send spans and evaluation results through Datadog's HTTP API.
Read more
Auto-instrumentation covers common model providers, including OpenAI and Azure OpenAI, Amazon Bedrock, Anthropic and Google Vertex AI, and several agent frameworks; Python has the widest list and Java the narrowest. Datadog also states that it supports the OpenTelemetry GenAI semantic conventions. We check your stack against the current list before quoting.
Datadog documents its APM OpenAI integration as separate from LLM Observability, so where services already run Datadog APM, we agree which one each uses and keep one tagging standard.
Judging answer quality, including Arabic answers
A trace tells you what happened; an evaluation tells you whether it was good. Start with a few questions the business already cares about: did the assistant answer in the language it was asked in, did it stay on the approved topics, did it use the retrieved document, did it expose personal data.
For Saudi applications the language question matters. Datadog's Language Mismatch evaluator flags answers written in a different language from the user's question, which catches an Arabic customer receiving an English reply. Checks that depend on meaning, such as whether an Arabic answer is faithful to the source policy, are better written as custom LLM-as-a-judge evaluators and tested on real Arabic examples, because a judge prompt that works in English can behave differently in Arabic.
Prompt injection deserves its own design. Detection in the observability platform shows attempts after they happen; blocking them belongs in the gateway and the application. Our guide to prompt-injection protection covers the controls on both sides.
LLM observability and Kong AI Gateway: which answers what
The two are often confused, and most production estates use both. The gateway is a control point every model call passes through; observability is the record of what the application did around those calls.
| Question | Kong AI Gateway | Datadog LLM Observability |
|---|---|---|
| Which team may call which model | Decides and enforces it | Shows which models were called |
| How much each team may spend | Enforces token and cost limits | Reports token usage and cost per request and step |
| Personal data in prompts | Redacts it before the prompt leaves your network | Scans and redacts what reaches the traces |
| Why one answer was slow or wrong | Sees the model call only | Sees retrieval, tool calls, agent steps and evaluations |
| Whether quality is improving | Not its job | Evaluations, topic patterns and user feedback over time |
Kong documents usage records with token counts, latency and cost through audit logs, metrics exporters and OpenTelemetry. We design how those records reach your observability platform for the versions you run. See our Kong AI Gateway page for the control side.
Prompts, personal data and residency in Saudi Arabia
Prompts and answers carry whatever users type: names, mobile numbers, national ID or Iqama numbers, account details. Unlike metrics, they are free text. Datadog is a SaaS platform, and as of November 2026 its documented sites are in the US, Germany, Japan, Australia and the UK, none in the Middle East. Decide which applications may send full prompts and answers, which send them redacted, and which send only metadata such as latency, tokens and cost.
Read more
Saudi organisations weigh this against the Personal Data Protection Law, overseen by the Saudi Data and AI Authority (SDAIA), and their own data classification. Interkey's engineers in Riyadh work through this with your security and data protection teams in Arabic or English, and our Datadog data residency guide sets out the options.
RelatedSaudi Data and AI Authority (SDAIA)Datadog data residency guide
What drives the cost
Datadog's documentation states that LLM Observability is metered and billed on ingested LLM spans, where an LLM span is a single request to a model provider. One agent run can therefore produce several billable spans. Cost follows request volume, how many model calls each request makes, and whether every environment is traced or only production. Custom LLM-as-a-judge evaluators also call a model, so we confirm how those calls are billed when the quote is prepared. List prices are on our Datadog pricing guide.
How an Interkey LLM observability project runs
-
Pick the first application.
One GenAI application with real users, and the questions the business wants answered about it.
-
Quote the licences.
Datadog licences sized on expected LLM spans, together with implementation and technical support.
-
Agree the data rules.
What may be sent in full, redacted or as metadata only, agreed with security and data protection.
-
Instrument.
Auto-instrumentation where the stack is supported, custom spans for workflows, agents and tools.
-
Design the evaluations.
Managed evaluators switched on, custom judges written and tested on Arabic and English examples.
-
Build dashboards and alerts.
Cost, latency and quality views per application, with alerts that name the step and the first thing to check.
-
Operate or hand over.
Interkey operates the platform from Riyadh, or hands it over with documentation and stays on as technical support.
Questions buyers ask
How is LLM observability different from the APM we already run?
APM shows where time went across services. LLM observability adds what only a GenAI application has: prompts and answers, retrieval and tool steps, tokens, cost and evaluations of answer quality. Most teams run both, with one tagging standard.
Which model providers can Datadog trace without code changes?
Datadog's auto-instrumentation covers providers including OpenAI and Azure OpenAI, Amazon Bedrock, Anthropic and Google Vertex AI, plus several agent frameworks. Coverage is widest in Python. Anything outside the list is traced with custom spans or the HTTP API.
Will our prompts and AI answers be stored outside Saudi Arabia?
What you send is stored on your Datadog site, and as of November 2026 none is in the Middle East. You can send prompts redacted, or send only metadata such as latency, tokens and cost, for applications whose data must not leave.
How is Datadog LLM Observability licensed and billed?
Datadog documents billing on ingested LLM spans, one span per request to a model provider. Interkey sells Datadog licences in Saudi Arabia and quotes them with implementation and technical support in one offer.
Do we need Kong AI Gateway as well as LLM observability?
Not to start. Observability explains what an application did; a gateway controls which models teams may call, how much they spend and what data leaves. Organisations with several teams calling models usually end up with both.
How long does it take to get a first AI application observed?
As an estimate, a few weeks for one application on a supported stack. What drives it: custom spans for unsupported frameworks, agreeing the data rules, and writing and testing evaluators in Arabic and English.
Next
Read next
Next step
Plan LLM observability for your AI app
Tell us which GenAI application you want to observe, what it is built with, which models it calls, and what data its prompts carry. We can quote the Datadog licences, the set-up and technical support together.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia