Observability
Most observability evaluations compare feature lists and then get decided by the invoice eighteen months later. This guide works through the four questions that actually determine the outcome — what you are trying to see, what drives the bill, where the telemetry lands, and who operates the thing afterwards — before any product name comes up.
Evaluation guide Published 6 min read
01Before the shortlist
An observability evaluation that begins with a shortlist is already committed to whatever the shortlist has in common. Every platform on it will collect logs, metrics and traces; every demo will look impressive; and the differences that decide whether the project succeeds are not the ones a demo shows.
The four questions below are the ones we have seen decide the outcome. They are worth settling in writing before a vendor conversation, because a vendor cannot answer them for you and will, reasonably, answer them in a way that suits their product.
02The four questions
None of these is a feature comparison. All four are decisions about your own environment that a product cannot make for you.
"Better visibility" is not a requirement. Name the three questions you currently cannot answer during an incident — which release caused this, which dependency is slow, is this one customer or all of them — and evaluate against those.
Observability pricing is usually a function of data volume, host count, or both, and the volume is generated by your systems rather than chosen by you. Model the bill against next year's log volume, not this month's.
Telemetry contains more than metrics. If logs carry request bodies, identifiers or payloads, the residency question is a data question rather than a hosting preference — and it is easier to answer before a contract than after.
A platform nobody owns decays into dashboards nobody reads. Decide who tunes the alerts, who removes the noisy ones, and who is allowed to say a signal is not worth collecting.
03The three shapes
| Model | What it buys | What it costs | Best when |
|---|---|---|---|
| Vendor SaaS | Fastest to value; no platform to operate; features arrive without a project | Data leaves your control; the bill scales with your data; residency is whatever the vendor's region list allows | Speed matters more than control, and no rule constrains where telemetry lives |
| Self-hosted open source | Full control of data and cost; no per-gigabyte pricing | You now operate a distributed system; storage, upgrades and on-call are yours | Residency or volume makes SaaS untenable, and you have platform engineers to spare |
| Split: local collection, selective export | Sensitive data stays put; the vendor still sees what it needs to be useful | Two things to run instead of one; the split has to be designed and defended | A rule constrains some data but not all of it — the common case |
04Before you shortlist
Work through these before a vendor conversation. Anything you cannot answer is a question to answer internally, not one to ask a salesperson.
A caution
A PoC answers one question, or it answers none
A proof of concept scoped as "try it and see" produces a demo everyone liked and no decision. A PoC is useful when it has a single question with a falsifiable answer: can this platform ingest our log volume within the budget, or can it correlate a trace across these three services without us rewriting them.
Write the question and the pass mark before the trial starts. If the answer would not change the decision, the PoC is a demo with a longer calendar entry.
05Writing it down
| Instead of | Write | Because |
|---|---|---|
| "Full-stack visibility" | "We can attribute a slow checkout to a specific service within five minutes" | The second can be demonstrated in a trial. The first can be claimed by everyone |
| "Reasonable cost" | "Under X per month at 400GB of logs a day, and we know what happens at 800" | Cost is a function of volume, and volume is the thing that grows without a decision |
| "Data stays in the Kingdom" | "These three log sources may not leave; everything else may" | The first rules out every SaaS. The second is usually what the rule actually says |
| "The team will own it" | "Named person, half a day a week, reviewing alert noise monthly" | Ownership without hours is a name on a slide |
Seen repeatedly
None of these is a bad product decision
Collecting everything because storage looked cheap. It was cheap at the volume being generated in month one. Ingest pricing compounds with growth that nobody authorised, and the review usually happens after the renewal rather than before it.
Migrating dashboards one for one. A dashboard set accumulated over five years contains a great many panels nobody has looked at since the incident that prompted them. Porting all of them moves the clutter and doubles the migration.
Treating alert count as a measure of coverage. A platform with four hundred alerts and a team that mutes most of them has less coverage than one with forty that are all acted on. Count the ones that were acted on, not the ones that fired.
06Buyer questions
Direct answers to the questions that come up in real evaluations. Anything missing, ask us at the bottom of the page.
Usually yes, but not for the reason most often given. The saving is rarely the licence — it is that one platform means one place to look during an incident, and correlation across signals stops being a manual step. The exception is a genuinely separate estate with its own team and its own on-call: forcing those onto a shared platform buys a shared bill and no shared understanding.
Less than is currently kept, almost always — but the honest answer is that nobody knows until they look. The useful exercise is to sort log sources by volume, then ask of the top five: when did anyone last query this, and what decision did it inform? Sources that fail both questions are candidates for sampling or archiving rather than indexing.
It depends on the model rather than the product. A self-hosted stack keeps everything in whatever data centre you run it in. A vendor SaaS puts data in the vendor's regions, and the region list is a published fact worth checking rather than assuming. A split design keeps constrained telemetry local and exports the rest, which is the common answer when some but not all of the data is constrained. Where observability telemetry actually lands works through the three in detail.
Only if there is a question it can answer that reading cannot. Ingest cost at your real volume, and correlation across your specific services, are good PoC questions. "Do we like the interface" is not: everyone likes the interface during a trial, and nobody is evaluating it at 3am six months later.
From the insights library
Question three of the four above, worked through properly: what telemetry contains, and what each deployment model does with it.
Buyer guideQuestion two, in detail. Ingestion, indexing, retention and the difference between them.
ComparisonWritten to say when it is, too. Useful once the four questions above have narrowed the field.
Continue exploring
Next step
If you have worked through the checklist and want a second read on it, describe the estate and the constraint — that is the conversation, rather than a demo.
Or directly
+966-11-2180999 info@interkey.com.saTawuniya Towers, North Tower, 7th Floor, King Fahad Highway, Olaya, P.O. Box 56835, Riyadh 11564, Saudi Arabia