The Live Incident Drill: Interviewing for How Someone Behaves When Prod Is Down

Every SRE interview asks some version of "tell me about a time production went down." Every candidate has a rehearsed answer where they stayed calm, found the root cause, and led the postmortem. The story is polished because they have told it ten times.

It tells you nothing about how they behave when the pager actually goes off in front of you.

Incident response is a behavior, not a fact. The traits that matter (staying methodical while a graph is red, communicating while investigating, resisting the urge to change everything at once) only show up under mild real pressure. So create some.

The setup: a 30-minute simulated outage

Give the candidate a running system in a real environment, then break it while they watch, or hand it to them already broken with a realistic alert: "the API is returning 500s for about half of requests, error rate started climbing ten minutes ago, find out what is happening and get it back."

A good incident scenario has a few properties:

• A believable symptom. Elevated error rate, latency spike, a service that is up but returning wrong data. Not a syntax error, a system-level fault.
• A findable cause with a real trail. Logs, metrics, or recent config that lead somewhere if you read them. The candidate should be able to reason their way in, not need to guess.
• A bounded scope. One or two services, so 30 minutes is enough to make real progress.
• A clear "restored" state. Everyone agrees when the incident is over.

Good causes to seed: a bad config rollout that a rollback fixes, a dependency that started timing out, a resource exhaustion that builds under load, a recent change that broke one code path. Each forces the candidate to observe, hypothesize, and act, rather than recite.

What you are actually measuring

The fix is almost secondary. You are watching for the incident-response behaviors that decide whether someone is safe on your rotation:

• Do they stabilize or investigate first? Strong responders often reach for the fastest safe mitigation (roll back, shed load, fail over) before chasing root cause. Getting the bleeding stopped is a real instinct.
• Do they communicate while working? In a real incident, silence is a failure mode. A candidate who says "I am seeing elevated 5xx on the API, checking whether the last deploy is involved, will know in two minutes" is showing you incident-comms muscle.
• Do they read before they act? Same as any debugging task: confirm the symptom, look at what changed, form a hypothesis. Panic looks like restarting everything and hoping.
• Do they change one thing at a time? And do they keep track of what they changed so they can undo it?
• Do they know when they are done? Verifying recovery, not just assuming the fix worked because the error stopped scrolling.

The anti-signal is someone who goes quiet, starts editing configs at random, and cannot tell you at the end what actually broke or why their change fixed it.

Scoring calm under fire without guessing

"They seemed calm" is not a score. Break it into observable moments so two interviewers agree (see how to write a technical interview rubric):

• Did they mitigate or at least consider mitigation before deep root-cause work?
• Did they narrate their state clearly enough that you could have jumped in?
• Did they localize the cause using evidence, or by guessing?
• Was the intervention reversible and minimal?
• Did they confirm recovery and could they summarize the incident at the end?

A candidate who does not fully resolve the outage but stays methodical, communicates, and correctly identifies the cause is a stronger hire than one who stumbles into a fix and cannot explain it. On a real rotation, the first person is the one you want holding the pager.

Why a real environment matters here

You cannot run an incident drill on a whiteboard. The pressure, the messy signals, and the temptation to over-react only exist when there is a real system responding to real commands.

EasyEnv runs the candidate in a live workspace with a seeded fault and records the whole session. The recording is the debrief: you and the panel watch the actual order of investigation, the commands, and the moment they either found the trail or lost it. It turns "I think they handled pressure well" into a shared, reviewable record. That is also what makes the debrief converge instead of drift (more on that in the interview debrief that reaches a decision).

The takeaway

Anyone can tell a good incident story. Far fewer can run an incident in front of you and stay methodical, communicative, and reversible under pressure.

Stop asking about the outage. Run one, and watch.