Observability
Kubernetes monitoring in Saudi Arabia, designed like it will be audited
The vendor documentation for monitoring a Kubernetes cluster is excellent and mostly answers the wrong question. Installing an agent takes an afternoon; getting it approved by a Saudi enterprise security team, sized so it does not bankrupt anyone, and operated through two years of cluster upgrades is the actual work. That is what this guide covers.
Implementation guide Published 5 min read
01Why now
What breaks without cluster visibility
Kubernetes is the most common trigger for the whole observability conversation, because it quietly invalidates the monitoring approach that came before it. Workloads no longer sit still on named hosts: they move between nodes, scale in and out, and leave no console to log into after the fact. The old model, an agent on a known server, watching known processes, stops describing reality the day the first Deployment ships.
The failure is rarely dramatic. It looks like an incident where nobody can say which pod served the failing requests, a capacity decision made on node averages that hide one starving namespace, and a platform team discovering that the cluster’s own components, the scheduler, the control plane, etcd, were never watched at all because every dashboard belonged to an application.
02First decision
How the agent gets in: three install paths and what each costs later
All three are legitimate; they differ in who does the ongoing work. The mistake is choosing by whichever tutorial came up first, because the choice is really about upgrades and governance, not installation.
| Path | What it is | The operational consequence |
|---|---|---|
| Operator-managed | A Kubernetes operator owns the agent’s lifecycle from a single custom resource | The most self-maintaining option and the recommended default for fleet-scale estates: upgrades become a resource edit, and configuration drift across clusters largely disappears. |
| Helm-managed | The agent ships as a chart your existing GitOps or CI flow owns | Fits teams that already manage everything as charts; the agent upgrades like any other release, and the same review gates apply, which security teams tend to like. |
| Hand-rolled DaemonSet | Manifests written and applied directly | Maximum control, minimum help: every upgrade, toleration and RBAC change is yours to maintain. Defensible in locked-down estates; a liability where nobody owns it. |
03The gate
The security review, prepared for instead of survived
In a Saudi enterprise the monitoring agent is a privileged workload arriving into a controlled environment, and the security review is where rollouts stall. These are the questions it will ask; arriving with written answers converts weeks into days.
- What does the agent access on each node: which mounts, which APIs, which privileges, and can it run unprivileged with reduced features where policy demands
- Exactly which outbound endpoints and ports it needs, as a list the firewall team can own and review on upgrade
- How it behaves behind the proxy: no direct egress from nodes, and nothing that silently bypasses it
- What RBAC it is granted inside the cluster, and why each permission exists
- What data leaves: which telemetry classes, with what filtering and redaction, per the estate’s residency rules
- How images arrive: from which registry, scanned by what, and mirrored where for clusters with no internet path
- The upgrade and rollback procedure, written for a change board: waves, health checks, and the trigger that reverses a bad rollout
04Disconnected estates
Air-gapped and private-registry clusters
A meaningful share of Saudi enterprise clusters cannot pull images from the internet, by policy or by network fact, and the vendor documentation lists the components without ever assembling the procedure. The workable pattern: mirror the agent images and charts into the estate’s own registry as a versioned, scanned artefact set; pin versions explicitly rather than tracking latest; and route telemetry out through the proxy or an aggregating collector tier in a controlled network segment, so nodes themselves never need an internet path.
The same design serves the sovereignty case: a cluster that must stay in-Kingdom gets its control point in front of any egress, and what may not leave, does not. None of this is exotic engineering; all of it needs to be decided before the rollout rather than discovered by it.
The bill, at cluster scale
The cardinality questions
Common question
Why do Kubernetes estates produce surprise observability bills?
Because tags multiply. Every unique combination of tag values on a custom metric is a billable series, and cluster estates generate combinations naturally: pod names, replica hashes, ephemeral job IDs. One high-cardinality tag on one popular metric can mint tens of thousands of series nobody intended. The fix is a written tagging standard and a cardinality budget per team, reviewed monthly like any other budget. The cost guide covers the log half of the same physics.
Common question
Do we keep Prometheus?
Usually yes, at least for a while, and the design should say how the two coexist rather than pretending one replaces the other on day one. Scraping existing exporters into the platform preserves the investment; what deserves scrutiny is double-collection, paying twice to move the same series, which coexistence designs produce when nobody looks.
05Day two
Operating cluster visibility through upgrades
Kubernetes versions arrive on a fixed cadence whether anyone is ready or not, and the monitoring layer has to survive every one of them. That means the agent upgrade is a rehearsed procedure with waves and a rollback trigger, not an evening’s optimism; it means someone reviews what each platform upgrade changes about the telemetry; and it means new namespaces and services inherit instrumentation and tagging standards by default instead of by memory.
This is the join between two Interkey practices: the container platform work builds and operates the cluster, and the observability work keeps it visible, on the Saudi working week, from Riyadh. The same buyer, three months apart, and designing the two together is cheaper than retrofitting either.
Running a cluster you cannot fully see, or planning one that a security review stands between you and? Interkey runs a cluster observability assessment: instrumentation gaps, a tagging and cardinality standard, and the security review pack, written for your estate.
06Buyer questions
What buyers ask us
Direct answers to the questions that come up in real evaluations. Anything missing, ask us at the bottom of the page.
Does this apply to managed Kubernetes, or only self-managed clusters?
Both. Managed control planes remove some operational work but none of the visibility questions: workloads still move, tags still multiply, and the security review still asks what the agent touches. Self-managed and sovereign clusters add the air-gap and residency layer on top, which is where Saudi estates differ most from the tutorials.
Can the cluster be monitored without cluster-admin level access?
The agent needs defined, reviewable permissions, not blanket admin, and modern agents document reduced-privilege modes with stated feature trade-offs. The honest framing for a security team: here is the permission list, here is why each exists, here is what turning each off costs in visibility. That conversation goes well when it is prepared and badly when it is improvised.
What should exist before go-live, besides dashboards?
A written tagging standard, a cardinality budget with an owner, the firewall endpoint list under change control, the agent upgrade runbook with a rollback trigger, and alerts on the cluster’s own components, the parts every application dashboard forgets. If a rollout ends without those artefacts, it is a demo that shipped.
From the insights library
Continue exploring observability
Controlling Datadog log costs
The cluster’s cost physics in full: what drives the bill and the levers that move it.
Technical guideObservability data residency in Saudi Arabia
For sovereign and in-Kingdom clusters: where telemetry may land, and the control point that enforces it.
PracticeDocker and Kubernetes consulting
The platform half of this conversation: architecture, security posture and who holds the pager.
Next step
Make the cluster visible, and keep it that way
Say which distribution you run, roughly how many clusters, and whether the security review is ahead of you or behind you.
Or directly
+966-11-2180999 info@interkey.com.saTawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia
Elsewhere