September 22, 2026·5 min read
Ci Cd Skills AssessmentDevOps HiringHands On AssessmentReal World EnvironmentsRole Specific Assessment
Anyone can add a dashboard. Very few engineers can find a problem they cannot reproduce, and that is the skill your on call rota actually runs on.
Observability interviews almost always become tool interviews. Have you used Prometheus. Do you know OpenTelemetry. What is the difference between a metric, a log and a span. The answers sort candidates by which vendor their last employer bought, and tooling is the part of this job that is learnable in a fortnight.
The part that is not learnable in a fortnight is narrowing. This post is about testing that.
"What are the three pillars of observability?" Metrics, logs, traces. It is a vocabulary check, and a model answers it better than any human candidate.
"How would you monitor a service?" You get a list: latency, errors, traffic, saturation, some dashboards, some alerts. Correct, generic, and unconnected to any judgment about what matters for a specific system.
"Tell me about an incident you debugged." The best question of the three, and still a story told backwards from a known answer. Every step sounds inevitable once you know the cause.
None of them shows you a candidate in the state that matters: staring at a system that is misbehaving, with several plausible explanations and no way to reproduce it on demand.
One scenario does most of the work here.
Give the candidate a running service that is slow for about four percent of requests. Not slow overall. The p50 is fine, the p99 is terrible, and no user can tell you a reliable way to trigger it. They get the traces, the logs and the metrics that a real system would have, and no information about which layer is at fault.
Cause options that work well, pick one: a connection pool that saturates only under a particular concurrency; a downstream dependency with a slow tail of its own; a cache that misses for a specific class of key; garbage collection pauses; one replica of three that is on a degraded host; a retry storm that only starts once a timeout is exceeded.
You are watching for one thing above all others: do they narrow, or do they wander.
Narrowing looks like this. They form a hypothesis, name the one signal that would kill it, go and get that signal, and either eliminate the hypothesis or follow it. "If it were the downstream service, the slow spans would all have a long child span there. Let me look at three slow traces." Two minutes later the branch is alive or dead and the space has halved.
Wandering looks like this. Every dashboard open. A comment about each graph. Latency is up, errors are up a bit, CPU looks fine, memory looks fine, let me check the database. Twenty minutes later they know a great deal about the system and have eliminated nothing.
The difference is not intelligence and it is not tool knowledge. It is whether they hold one hypothesis at a time and test it.
Do they look at the slow requests specifically? The single most common failure is reasoning about aggregates. A p99 problem is invisible in an average, and a candidate who spends the session on mean latency has not internalised that the interesting requests are a minority by definition. Reaching for a trace filtered by duration in the first few minutes is a strong positive.
Do they compare against something? A slow endpoint is only slow relative to something: yesterday, the other replica, the other endpoint, the other customer. Candidates who establish a baseline are much faster and much more often right.
Do they notice what is missing? Sometimes the honest answer is that the system cannot be diagnosed from what it currently emits. An engineer who says "there is no span around the cache lookup, so I cannot tell from here, and that is the first thing I would add" has given you a better answer than a lucky guess. Note whether they say it at minute four or minute twenty-five.
Do they ask what "acceptable" is? Four percent slow may be an emergency or may be fine. Engineers who ask about the SLO before optimising have worked somewhere with one.
Do they resist the fix? There is a strong pull to start changing configuration. Pool size, timeouts, replica count. A candidate who tunes before diagnosing will do the same to your production system, and the change will sometimes appear to work, which is worse.
Traces and logs are not props. You cannot hand a candidate a screenshot and learn whether they would have filtered that view themselves.
In EasyEnv the candidate gets a real environment with the service, its dependencies, a load generator producing the tail, and the observability stack wired up. They can query it, change it, and re-run. The session is recorded, which matters here more than in most assessments: the order in which they opened things is the assessment, and it is almost impossible to reconstruct from memory afterwards.
The same setup lets you be relaxed about AI. A model is genuinely useful for "what does this GC log line mean" and useless for "which of these three things is happening in this cluster right now", because it cannot see the cluster. Watch which kind of question the candidate asks it. That distinction is the practical core of AI literacy for engineers.
Knows the tools. Can build a dashboard, write a query, explain the pillars. Useful, and not yet diagnostic.
Narrows. Hypothesis, signal, eliminate, repeat. Looks at slow requests specifically, establishes a baseline, knows when the data cannot answer the question. This is what you are hiring.
Has owned the pager. All of that, plus the second half of the job: what alert would have caught this sooner, what would have fired instead, what is the cost of the instrumentation they are proposing, and which existing alert should be deleted because nobody acts on it. Alert fatigue is a real operational skill and almost nobody interviews for it.
Do not require the right answer. Some of these causes are genuinely hard. A candidate who eliminates three hypotheses cleanly, states the fourth, and runs out of time has outperformed one who guesses correctly in minute two.
Do not use a vendor the candidate has never seen and then judge their speed. Tell them which stack it is in advance, or accept that the first five minutes are orientation.
Do not separate this from debugging skill generally. It is the same muscle at a different altitude: how to assess debugging skill and the database performance interview are the same test aimed at other layers.
When something last went wrong for a small fraction of your users, how long did it take to find out why?
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Ask a candidate to describe a good CI pipeline and everyone gives the same clean answer. Hand them a build that passed yesterday and fails today, and you find out who has actually lived with one.
The best Linux interview question is not a question, it is an SSH prompt on a machine where something is wrong. Here are six scenarios, what each one separates, and how to read the first ninety seconds.
Read more
Timed task platforms measure correct code written alone against a clock, which is a real skill and a narrow slice of the job. Here is what a task score can and cannot see, and how to decide whether you need a different kind of assessment.
Read more
