Datadog · Monitoring alerts
Monitoring alerts in Saudi Arabia: fewer alerts, sent to the right person, on the right channel
Monitoring alerts are only useful if someone acts on them. Most teams have hundreds of notifications a week, many about the same fault, and the one that mattered lost among them. This guide explains why alert fatigue happens, which Datadog monitor settings reduce it, how alerts reach the on-call engineer, when an SMS is the right channel in Saudi Arabia, and where Datadog fits and where it does not.
Published 9 min read
Orientation
At a glance
The whole page in one table, before the detail.
- The problem
- Too many alerts, too little meaning; people stop reading them
- What fixes it
- Fewer monitors, on symptoms, with sensible thresholds, grouping and regular review
- In Datadog
- Monitors: a query, a condition and a notification, with warning, recovery, no-data and renotify options
- Routing
- Notification rules by tag, team handles, and Pages to a Datadog On-Call team
- SMS alerts
- Through On-Call where Datadog supports the country, or a webhook to Interkey Digital Connect on Saudi networks
- What Interkey does
- Alert review, monitor redesign, routing and SMS set-up, licences and support from Riyadh
The problem in plain words
- The usual situation: a shared mailbox or chat channel full of alerts, most of them closed without action, and a real outage reported first by a customer.
- Why it happens: monitors were added one at a time, on every metric that had a number, with thresholds nobody revisited after the traffic grew.
- What changes it: fewer alerts, each tied to something a user would notice, sent to the team that owns the service, with the first thing to check in the message.
- The tool's part: Datadog gives the settings; what deserves a person's attention is your decision.
Monitoring alerts: why most of them get ignored
Alert fatigue is what happens when people receive more alerts than they can act on. After weeks of false alarms an engineer treats every notification as one that can wait, and the real alert is read an hour late.
The causes repeat. Monitors watch CPU on every host rather than the errors and slow responses users feel. A threshold set for last year's traffic fires every afternoon. One database fault fires twenty monitors. Maintenance pages the on-call engineer at night. And alerts go to everybody, so nobody owns them.
This is unfinished design, not a tooling failure. Our Datadog implementation page sets out four tests every alert should pass; this guide applies them in the monitors.
Why alerting matters for teams in Saudi Arabia
Banks, government entities and retailers in the Kingdom run services used late at night, through Ramadan and on salary days. Alerting built around office hours misses that, and on-call duty has to fit a Sunday-to-Thursday week with weekend cover.
Read more
Three local questions come up early. First, which phones the alerts reach: SMS and voice from Datadog On-Call depend on Datadog's list of supported countries. Second, what an alert may contain: a message that quotes a customer's mobile number or national ID sends personal data to every recipient, so templates should name the service and the symptom, not the customer. Third, records: the National Cybersecurity Authority (NCA) Essential Cybersecurity Controls include logging and monitoring of systems,. The authority's own text is the reference.
Alert fatigue: its common causes and the Datadog monitor settings that address them
Each setting below is documented by Datadog. Choosing the values is the design work.
| Cause of noise | Datadog setting | What it does |
|---|---|---|
| A spike of one minute pages someone | Evaluation window | Judges the condition over a rolling window: 5 minutes, 15 minutes, 1 hour, or a custom window of up to one month |
| The alert flaps between alert and OK | Recovery thresholds | Require the value to come back further than the alert threshold before the monitor recovers |
| Everything is urgent | Warning threshold | Sends a warning before the alert threshold, which can go to a channel instead of a phone |
| One alert per host when one would do | Simple alert, or notification grouping | A simple alert aggregates all sources into one notification; dimensions left out of a multi alert become sub-groups, so fewer notifications go out |
| Cloud metrics arrive late and look like a drop | Evaluation delay | Waits before evaluating; Datadog recommends 15 minutes for cloud metrics |
| A missing metric is read as a failure, or ignored | No-data options | Choose to evaluate as zero, show the last state, show NO DATA with or without a notification, or show OK |
| Maintenance pages the on-call engineer | Downtimes | Silence notifications, once or on a recurring schedule, by monitor name or by tags |
| A rule needs two conditions together | Composite monitor | Combines up to 10 monitors with AND, OR and NOT, and alerts only when the combined condition is true |
Where a fixed threshold cannot fit the traffic pattern, anomaly monitors and alert correlation take over, as our AIOps page explains.
Datadog monitors: building an alert that means something
In Datadog an alert is a monitor: a query, a condition and a notification. Datadog documents monitor types for metrics, hosts, logs, APM, synthetic tests, SLOs and more. Three habits make them quieter.
Alert on the symptom, warn on the cause
Page a person for what users feel: error rate, latency, a failed synthetic test of a login or payment journey, a queue that will not drain. Keep disk and memory at warning level, sent to a team channel, unless they predict a failure within hours. Our SLO monitoring guide shows how service level objectives turn this into a target the business agrees.
Read more
Write the message for the person reading it at night
A Datadog notification message is a template. Conditional blocks such as {{#is_alert}} and {{#is_warning}} change the text by state, tag variables such as {{host.name}} fill in where the problem is, and a {{#is_renotify}} block carries an escalation message for an alert still open. A good message names the service and symptom, links the dashboard and runbook, states the first check, and quotes no customer data.
Repeat only what is still unresolved
Renotification resends an alert at an interval you set, for the states you choose (alert, warn, no data), and can stop after a set number of repeats.
On-call alerts: routing each alert to the team that owns it
An alert that goes to everyone belongs to no one. Datadog sends notifications to recipients named with an @ handle in the message: a Datadog user's email, an external email address, a Slack or Microsoft Teams channel, a Jira or ServiceNow integration, a webhook, or a team handle that forwards to the team's own channel.
Handles in hundreds of messages do not scale. Datadog recommends monitor notification rules, which add recipients from the monitor's tags, so every monitor tagged with a team reaches that team. For Slack, a rule can thread one alert cycle under one message.
Read more
For alerts that need a person now, the notification goes to a Datadog On-Call team and becomes a Page. The team's routing rules set its urgency, choose the escalation policy and can hold escalation until support hours. Schedules, escalation and incidents are on our incident management page.
Relatedincident management page
SMS alerts on Saudi mobile networks
Push notifications from the Datadog mobile app and email reach an on-call engineer anywhere. SMS suits people outside the rota who must know about a major outage, such as a duty manager or a service owner, and engineers whose app notifications are unreliable at night.
- Through Datadog On-Call. Each responder can add SMS and phone calls as contact methods. Datadog states that phone number support varies by country, so check your engineers' numbers first.
- Through Interkey Digital Connect. Interkey Digital Connect, Interkey's SMS platform, takes transactional alerts through a REST API, with sender identities registered on the Saudi mobile networks. Datadog's Webhooks integration calls an HTTP endpoint with a payload you define, so a webhook from chosen monitors to Digital Connect sends an SMS to a list you control.
Keep SMS for the few monitors that justify it. Our Datadog integrations page covers webhooks.
An alerting review checklist
Questions to ask of any alerting set-up, including one we built.
| Question | Why it matters |
|---|---|
| Which monitors fired last month with no action taken? | They are the first to change or remove |
| Which incidents did a user report before any monitor? | They show the alerts that are missing |
| Does every monitor that pages have an owning team? | Unowned alerts are read by nobody |
| Does each message say what to check first? | An alert that starts from zero costs time |
| Which monitors are long muted, or stuck in alert? | Datadog's Monitor Quality page lists both |
| Who reviews alert quality, and how often? | Thresholds drift as traffic grows |
Datadog's Monitor Quality page flags seven kinds of problem, among them a high volume of alerts, muting for over 60 days and missing evaluation delays on cloud data.
Where Datadog fits, where it does not, and what drives the cost
Datadog alerting fits when the data behind the alerts is already in Datadog: metrics, traces, logs, synthetic tests and user sessions, with the same service tags. Then one monitor links the dashboard, trace and logs an engineer needs.
Where it may not fit: systems that cannot send data to a SaaS platform keep their own alerting. Datadog is SaaS, and as of November 2026 none of its documented sites is in the Middle East; our Datadog data residency guide sets out what that means. And no tool fixes alerting nobody owns.
Read more
What drives the cost of alerting
Monitors read data that Datadog already bills for, so the cost of alerting follows the products in use: hosts, APM, logs and synthetic tests. On-Call is billed per seat for the people who take Pages, as our incident management page explains. SMS through Digital Connect is quoted by traffic mix and volume. List prices are on our Datadog pricing guide; Interkey quotes licences, set-up and support together.
As an estimate, an alert clean-up takes weeks. What drives it: the number of monitors and services, tag consistency, and how many teams must agree on ownership.
RelatedDatadog data residency guide
How Interkey runs an alerting review
-
Read the alert history.
What fires most, what nobody acts on, what no monitor caught.
-
Agree ownership and tags.
A team and a service on every monitor, so routing follows the tags.
-
Redesign the monitors.
Symptoms first, warning and recovery thresholds, windows and delays that fit your traffic.
-
Set the routes.
Notification rules, team channels, Pages to On-Call teams, and SMS for the few alerts that need it.
-
Silence planned work.
Recurring downtimes for maintenance windows, agreed with change management.
-
Test with a drill.
A test alert through every route, checked on real phones before go-live.
-
Operate or hand over.
Interkey, an official Datadog partner in Saudi Arabia, sells the licences, implements and supports them from Riyadh, or hands over with documentation.
Questions buyers ask about monitoring alerts
How do we reduce alert fatigue without missing a real outage?
Remove or downgrade monitors nobody acts on, and page only for symptoms users notice: errors, latency, failed journeys. Evaluation windows and recovery thresholds stop short spikes paging. Then add monitors for recent incidents none caught.
How does a Datadog monitor decide which on-call team gets the alert?
Through the recipients in its message or, better, notification rules that add recipients from the monitor's tags. Sent to a Datadog On-Call team, the alert becomes a Page, and the team's routing rules set its urgency and escalation policy.
Can a duty manager outside the on-call rota get critical alerts by SMS in Saudi Arabia?
Yes. A webhook from chosen monitors to Interkey Digital Connect sends an SMS on Saudi mobile networks, with a registered sender identity, to a list you control.
How do we stop alerts during planned maintenance?
Schedule a Datadog downtime, once or on a recurring schedule, scoped by monitor name or tags. Monitors keep evaluating but send no notifications. If a monitor is still alerting when the downtime ends, Datadog notifies by default.
What does alerting cost in Datadog for a company in KSA?
The cost follows the Datadog products whose data the monitors read, On-Call seats for the people who take Pages, and SMS traffic if you use Digital Connect. Interkey quotes licences, set-up and support together.
How long does it take to clean up our existing monitors and alerts?
As an estimate, weeks rather than months for most estates. It depends on the number of monitors and services, how consistent your tags are, and how quickly teams agree who owns what.
Next
Read next
Next step
Review your monitoring alerts with Interkey
Tell us roughly how many alerts your teams receive a week, where they arrive and who is on call. We reply with what we would remove, add and reroute.
Or directly
+966-11-2180999 [email protected]Tawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia