November 2, 2026·5 min read
A checkout service starts throwing errors for about one in twenty requests at 2 a.m. By the time anyone looks, three things have already happened in the last hour: a feature flag rolled out to 5% of traffic, a TLS certificate rotated on the load balancer, and the nightly reconciliation job kicked off against the same database. Any one of the three is a defensible answer. Only one of them is the actual cause.
That gap, between a defensible answer and the confirmed one, is what a root cause analysis assessment is supposed to measure. Most interviews never build a scenario with more than one suspect, so they never find out whether a candidate can close it.
Root cause analysis is not the same skill as debugging and it is not the same skill as staying calm during an outage. Debugging asks "what do I change to make this pass." Incident response asks "how do we restore service and communicate while we do it." Root cause analysis asks a narrower question: given a symptom and several things that changed around the same time, which one actually produced it, and how do you know.
The "how do you know" is the part that gets skipped. Google's SRE Workbook is blunt about this in its postmortem guidance: it recommends avoiding the phrase "the root cause" altogether, because incidents are rarely explained by one factor and the label tends to make people stop looking once they have found a factor rather than the one that fits the evidence.1 That is the exact failure mode worth designing an assessment around: not whether a candidate can name a plausible cause, which almost everyone can, but whether they can rule out the other plausible causes before they commit.
Ask a candidate to describe a past root cause analysis and you get a story: clean, resolved, and edited by however many times they have told it since. The story format hides the one behavior you actually want to see, which is what they did between having a hunch and being sure.
It also fails for a structural reason. A story has one cause because the storyteller already knows the ending. A live scenario has to be built with more than one candidate cause, or it does not test discernment at all. If there is only one thing that changed before the symptom appeared, whoever notices the timeline wins, and you have measured attentiveness, not analysis.
Take the checkout example above and hand it to a candidate as three timestamped facts plus a real environment: logs, a metrics dashboard, and access to the affected service. Say what broke and when, and stop there.
A scenario like this only works if each decoy is genuinely plausible and genuinely checkable:
The design principle behind all three: every decoy needs its own piece of evidence that would confirm or kill it, and that evidence has to be reachable in the environment you hand over. If ruling something out requires information the candidate cannot get, you have built a guessing game, not an assessment.
Watch four things, and score them separately from whether they land on the right answer:
Did they generate more than one hypothesis before committing to one? A candidate who locks onto the feature flag in the first ninety seconds and spends the rest of the session building a case for it has shown you confidence, not analysis. The strong move is naming two or three suspects early and then spending the session eliminating them.
Did they look for evidence that could kill their favorite hypothesis, not just support it? This is the practical version of avoiding confirmation bias. Someone who says "if it's the flag, the non-flagged traffic should be clean, let me check" is testing their idea against a fact that could disprove it. Someone who only looks for facts that agree with their first guess will find them, because in a real system you can usually find something that correlates with anything.
Can they state the mechanism, not just the correlation? "It started around when the job runs" is a timing observation. "The job holds a lock on the same table for four seconds, and the requests that fail are the ones whose timeout is under four seconds" is a mechanism. Only the second one tells you the fix will actually prevent a recurrence, and only the second one would survive someone else checking their work.
Would their conclusion survive being wrong about the first two? A candidate who checked the flag, checked the certificate, and only then landed on the job has a conclusion you can trust, because you watched the other explanations get closed off. A candidate who jumped straight to the job and happened to be right has the same answer with none of the same confidence behind it. Score the elimination, not the destination.
This is close to the discipline behind how to assess debugging skill, but it is not the same test. Debugging can be scored on whether the fix works. Root cause analysis has to be scored on whether the reasoning would still hold up if you changed which of the three things had actually gone wrong that night, which is why the decoys matter more here than in almost any other assessment.
It is tempting to fold root cause analysis into a live incident simulation, and the live incident drill is a good assessment in its own right. But it is measuring something adjacent: communication under pressure, mitigation instinct, whether someone stays methodical while a graph is red. You can run a strong incident drill where the candidate restores service in ten minutes by rolling back everything at once and never actually learns which change was responsible. That is a good incident response. It is not root cause analysis, and treating the two as interchangeable is how teams end up with people who are excellent at making outages end and weak at making sure the same one does not happen again next month.
If you only have time for one exercise, decide first which behavior you are hiring for. If you have time for both, run them separately, because a candidate under outage pressure is optimizing for speed, and rushing a root cause analysis is exactly the shortcut you are trying to catch.
None of the four things above are visible on a whiteboard or in a written case study. "I would check whether the non-flagged traffic looks clean" is a sentence anyone can say. Whether a candidate actually goes and checks, and what they do when the answer is not what they expected, only exists when there is a real dashboard to query and real logs to read.
In EasyEnv the candidate works in a live environment seeded with the scenario, and the session is recorded end to end. The review is not "did they get it right." It is the order of operations: which hypothesis they checked first, whether they went looking for disconfirming evidence or only confirming evidence, and whether the mechanism they landed on is the one that actually explains the failed requests. That sequence is close to invisible in a debrief built from memory, and it is the entire signal in a recording.
It is also a fair test of AI use, which is worth being explicit about. A model will produce a competent-sounding explanation for any of the three decoys on request, complete with plausible mechanism language. What it cannot do is query your dashboard, notice that the non-flagged traffic is just as broken, and revise its answer. Let candidates use a model for this exercise. The elimination step is where the difference between a candidate who investigates and one who narrates shows up clearly.
You do not need a fictional checkout service to run this well. Take a real incident from your own history that had at least two plausible explanations at the time, even if you have since agreed on the real one. Seed a version of it with the same two decoys, keep the evidence that would have let someone rule each one out, and see whether a candidate reaches the same conclusion your team did, and by the same route.
What was the last incident on your team where the first explanation everyone reached for turned out to be wrong?
Google SRE Workbook, "Postmortem Culture: Learning from Failure," on avoiding the phrase "root cause" in favor of identifying multiple contributing causes: https://sre.google/workbook/postmortem-culture/ ↩
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Hire the first engineer and you want someone who can do everything. Hire the tenth and you want someone who makes nine other people faster. Most startups run the same loop for both. Here is what to test at each stage, and the failure each one prevents.
Rust interviews still argue about the borrow checker, which the compiler settled before anyone sat down. Here are four tasks on a service that compiles cleanly and behaves badly, panics in the request path, a blocked async runtime, errors that help nobody, and a hot path nobody measured.
Read more
The backup job is green and nobody has ever run the other half. Here are four tasks on a real box that test whether a candidate can restore, verify, and tell you in minutes how much data was lost, which is the only number that matters during an incident.
Read more
