Datadog · Kafka monitoring
Kafka monitoring in Saudi Arabia: brokers, consumer lag and end-to-end latency in Datadog
Kafka monitoring tells you whether the events your business runs on are arriving on time: whether brokers are healthy, how far each consumer group has fallen behind, and how long an event takes to travel from the service that produced it to the last service that reads it. Datadog reads broker metrics over JMX, consumer offsets from the brokers, and, with Data Streams Monitoring, the latency of each pipeline across topics and queues. Interkey sells Datadog licences in Saudi Arabia, sets up the integrations, builds the dashboards and alerts, and provides technical support from Riyadh.
In short
- What it covers: Kafka brokers, topics and partitions, consumer groups and their lag, and the services that produce and consume events.
- How data is collected: the Datadog Agent runs the Kafka check over JMX on each broker and the Kafka Consumer check against the brokers; Data Streams Monitoring adds pipeline latency from your producers and consumers.
- Also covered: Kafka on Kubernetes with Strimzi, Amazon MSK, and RabbitMQ and other message queues.
- Who it is for: platform and integration teams running event-driven systems, such as payments, notifications or data replication.
- What Interkey does: licences, Agent and integration set-up, tagging, lag and latency alerts, dashboards and technical support.
At a glance: what to watch in Kafka and where Datadog reads it
Kafka monitoring is several questions, each answered from a different place. Read together, they show whether a slowdown is in the broker, the consumer or the service upstream.
| Signal | Where Datadog reads it | What it answers |
|---|---|---|
| Broker health | Kafka check: JMX metrics such as under-replicated partitions and request latency | Is the cluster itself healthy and keeping replicas in step? |
| Consumer lag in messages | Kafka Consumer check: broker and consumer offsets per partition | How many messages each consumer group still has to read |
| Consumer lag in seconds | Data Streams Monitoring, from instrumented producers and consumers | How old the events a consumer is reading are |
| End-to-end latency | Data Streams Monitoring, across topics, queues and services | How long an event takes to cross the whole pipeline |
| Producers and consumers | APM traces and the Data Streams topology map | Which service writes to and reads from each topic |
| Hosts, JVMs and containers | The Datadog Agent and infrastructure monitoring | Whether the machines and pods under Kafka have the resources they need |
What Kafka monitoring has to cover
In banks and telcos, Kafka often sits between the systems that matter most: a payment is published as an event, a fraud check and a notification service consume it, and a ledger or data platform reads it again. When a customer says a transfer is slow, the broker can be perfectly healthy while one consumer group is an hour behind.
Read more
Good message queue monitoring therefore covers three layers. The cluster: brokers, partitions, replication and the hosts they run on. The consumers: how far behind each group is, in messages and in time. And the pipeline: the time from the first producer to the last consumer, across every topic in between. Broker metrics alone answer only the first; Datadog covers all three, linked by shared service and environment tags.
Broker metrics and Kafka consumer lag monitoring
Brokers. The Datadog Kafka check collects broker metrics over JMX, through the Agent's JMXFetch. Datadog notes that a JVM is needed on each Kafka node for this, and the JVM Kafka already runs on can serve. The metrics cover replication, the controller, the network and requests, including KRaft metrics. Datadog sets a limit of 350 metrics per check instance, so on a large cluster Interkey chooses which beans to collect rather than collecting everything.
Read more
Consumer lag in messages. The Kafka Consumer check reads the latest offset on each partition from the brokers and compares it with the offset each consumer group has committed. Its metrics include the lag per partition and consumer group. It ships with the Agent and runs from a host or container near your consumers, with nothing extra installed on the brokers.
Consumer lag in seconds. A lag of 10,000 messages is nothing for a busy topic and a serious delay for a quiet one, so an alert on message count alone is either noisy or late. Data Streams Monitoring reports lag in seconds between producer and consumer, which is easier to set a threshold on. Datadog documents this for Java services from version 1.9.0 of its Java tracer, and notes that kafka-clients 3.7.x, used by Spring Boot 3.3.x, does not support lag generation. Our infrastructure monitoring page covers the Agent on the hosts underneath.
Data Streams Monitoring: end-to-end latency across topics
Data Streams Monitoring follows an event from the service that produced it, through each topic or queue, to every service that consumes it. Datadog describes it as measuring pipeline health with end-to-end latency for events crossing your system, together with throughput in bytes and consumer lag. A topology map shows which producer and consumer services depend on which queue, and a monitor can be created on any pathway, for example from the payment API to the ledger service.
Read more
It works by propagating context through message headers. You turn it on with an environment variable, DD_DATA_STREAMS_ENABLED=true, on each service that sends or consumes messages, and the Datadog libraries do the rest; Datadog states that services already sending APM data through OpenTelemetry need no extra set-up. Datadog lists Java, Python, .NET, Node.js, Go and Ruby, and technologies including Apache Kafka, RabbitMQ, Amazon SQS, Amazon SNS, Amazon Kinesis, Google Pub/Sub, Azure Service Bus and IBM MQ.
Because it runs inside your services, Data Streams Monitoring goes with Datadog APM: Datadog includes it in APM Pro and APM Enterprise. Traces then show the request that produced an event, and the topology map shows where that event went next.
Strimzi monitoring, Amazon MSK and RabbitMQ monitoring
Kafka on Kubernetes with Strimzi. The Datadog Strimzi check reads the Prometheus metrics of the Strimzi Cluster, Topic and User Operators, from Agent 7.47.0. The brokers themselves are still monitored with the Kafka and Kafka Consumer checks, configured through Autodiscovery so that new broker pods are picked up as they start.
Amazon MSK. Datadog states that its Kafka check cannot be used with Amazon MSK. MSK has its own integration instead: the Agent, on a client machine, reads the Prometheus metrics MSK exposes once open monitoring is turned on, or a crawler reads MSK metrics from CloudWatch without the Agent.
RabbitMQ. The Datadog RabbitMQ check reads either the Prometheus plugin, which RabbitMQ enables by default from version 3.8, or the management plugin. It covers queues, messages published, delivered, acknowledged and redelivered, consumers, and node memory, disk and file descriptors.
Where teams in Saudi Arabia use it
- Payment and transfer events. An alert on lag in seconds for the consumer groups behind a payment flow, routed to the team that owns them, before customers notice a delay.
- Telecom usage and notification events. High-volume topics where throughput changes through the day, so latency thresholds work better than fixed message counts.
- Change data capture into Kafka. Database changes captured and published to Kafka for other systems to read. Our change data capture guide explains the pattern, and iblync's own website lists Apache Kafka among the systems it connects; see real-time replication with iblync. Monitoring then shows whether the target is minutes or seconds behind the source.
- Moving Kafka to Kubernetes. The same broker and lag view before and after a move to Strimzi, so a regression is visible on the first day.
Relatedchange data capture guidereal-time replication with iblync
Data residency and local support
Datadog is SaaS, and as of November 2026 its documented sites are in the US, Germany, Japan, Australia and the UK, none in the Middle East. The Kafka, Kafka Consumer and RabbitMQ checks send metrics about brokers, topics, partitions and consumer groups, not the content of your messages. Topic names, group names, tags and logs can still carry identifiers, so Interkey agrees with your security team what is collected, how topics are named in Datadog, and which log fields are removed before they leave your network. Our Datadog data residency guide covers the wider question. Support comes from Interkey's engineers in Riyadh, in Arabic and English.
RelatedDatadog data residency guide
How a Kafka monitoring project runs with Interkey
-
Map the pipelines.
Clusters, topics, consumer groups and the services on each side, starting with the flows the business depends on.
-
Quote the licences.
Infrastructure hosts for brokers, and APM with Data Streams Monitoring for the services that produce and consume, quoted with implementation and support.
-
Set up broker and lag collection.
JMX on the brokers, the Kafka and Kafka Consumer checks, or the MSK or Strimzi integration, reviewed with your platform team.
-
Turn on Data Streams Monitoring.
Service by service through your deployment pipeline, starting with the most important pathway.
-
Agree tags, then build.
One standard for service, environment, team and cluster; then lag and latency monitors that name the owner.
-
Review noise and cost.
Alert volume, collected metrics and log volume, after the first weeks in production.
-
Operate or hand over.
Interkey operates the platform from Riyadh, or hands it over and stays on as technical support.
What drives the cost and the timeline
The Datadog cost depends on the hosts running the Agent, including the brokers, the number of services that need APM and Data Streams Monitoring, the log data you index and keep, and how much each integration collects. Our Datadog pricing guide sets out how each product is billed; Interkey quotes the licences with the set-up and support.
Read more
The timeline is an estimate, not a promise. Broker metrics and consumer lag in messages usually arrive within days of JMX and network access being agreed. Data Streams Monitoring across the important pipelines typically takes a few weeks, because each service is changed through its own deployment. The drivers are the number of clusters, the client libraries and versions in use, and how quickly change approvals for production services are given.
Questions buyers ask
Can Datadog show Kafka consumer lag in seconds, not only in messages?
Yes, through Data Streams Monitoring, which reports lag in seconds between producer and consumer for instrumented services. The Kafka Consumer check reports lag in messages per partition and consumer group, without changing your services.
Do we have to install anything on our Kafka brokers?
For broker metrics, yes: JMX enabled and the Datadog Agent on each broker host, using the JVM Kafka already runs on. Consumer lag in messages is read from the brokers by an Agent elsewhere, with nothing added to them.
Can Datadog monitor Kafka on Strimzi or Amazon MSK?
Yes. On Strimzi, a Datadog check reads the operators and the Kafka checks read the brokers through Autodiscovery. Amazon MSK has its own Datadog integration, because the standard Kafka check does not work with it.
Does Kafka monitoring send our message content outside Saudi Arabia?
The Kafka checks send metrics about brokers, topics and consumer groups, not message content, to a Datadog site outside the Kingdom. Interkey agrees with your security team which names, tags and log fields are collected.
How is Data Streams Monitoring licensed for teams in KSA?
Datadog includes Data Streams Monitoring in APM Pro and APM Enterprise, billed per APM host. Interkey sells Datadog licences in Saudi Arabia and quotes them with set-up and support; our pricing guide has the list prices.
How long does it take to set up Kafka monitoring?
Broker metrics and lag in messages usually arrive within days once access is agreed. Data Streams Monitoring across key pipelines typically takes a few weeks, depending on clusters, client versions and change approvals.
Next
Read next
Next step
Map your Kafka pipelines in Datadog
Tell us how many Kafka clusters you run, where they run (servers, Kubernetes with Strimzi, Amazon MSK), which flows matter most, and whether Datadog is already in use. We can quote the licences, the set-up and technical support together.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia