August 10, 2026·6 min read
Ask a candidate whether they use AI in their work. They say yes. Everyone says yes.
Ask which tools. You get a list. Ask for an example. You get a story that sounds fine. Twenty minutes gone, and you know nothing you did not know from the CV.
The claim has stopped carrying information. Not because people are lying, but because "I use AI daily" now covers everything from a person who pastes an error into a chat box to a person who has genuinely changed how they work.
The only thing that separates them is watching. This post is what to watch for.
We started letting candidates use whatever they wanted in real-environment sessions, then reviewing the recordings. The differences between people show up fast, usually inside five minutes, and they are not subtle.
Four behaviors do the separating.
The first thing a strong candidate does is often not open a model.
They read the error. They run the thing. They look at the config. Then, if the task turns out to be a shape a model is good at, they use one.
Weaker candidates open the chat box first, every time, including for a task where the answer is one line down in the log file they never opened.
What good looks like: they can say why they did or did not use it here. "This is a config I have never seen, so I would rather read the docs than get a plausible guess." Or: "This is boilerplate, I am not typing it out."
The tell: watch the first ninety seconds. Reflex or decision?
A task that surfaces it: give them a failing service where the cause is visible in the logs, and a second task that is genuinely tedious boilerplate. A person with judgment treats them differently. A person without treats them the same.
This is the part people call prompt engineering. The skill is not phrasing. It is knowing what the model cannot see.
What good looks like: the first request already contains the constraint, the versions, the real error text, the convention the codebase uses, and what the answer should look like. The first answer comes back close, and they adjust once.
What weak looks like: a short request, a generic answer, then the same request reworded five times. Increasingly frustrated, no new information added. This is the single most common failure we see, and it costs people ten minutes at a stretch.
The tell: does the second attempt add facts, or just adjectives?
A task that surfaces it: any task inside a real codebase with its own conventions. Generic answers do not fit, so the person has to supply the local context or fight the output forever.
The heavy one. Generated code arrives confident, tidy and sometimes wrong.
What good looks like, in order: they check the functions actually exist in the version installed. They run it against the real case, not the test that came with it. They name the edges out loud (empty, null, big, concurrent) and try one. Then they read the error paths, which is where the weakest generated code lives.
What weak looks like: it compiles, so it is done. Or they run the model's own test, which asserts the behavior the model implemented, bug and all.
The tell: did the code get executed against something real before they moved on? That is a binary, and the recording answers it.
A task that surfaces it: hand them a pre-written pull request that a model produced, and ask what they would do with it. Plant a call to a function that does not exist, a dropped edge case, and one thing that is not a bug at all but does not fit the codebase. We go deep on this one in the code compiles, reads well, and nobody understands it.
The quiet one, and the one that turns into an incident.
What good looks like: they notice before pasting that the log contains a customer email or a live token, and they trim it. They flag that a suggested dependency has an unclear licence. They say plainly which part of the final answer they would not merge without someone else reading it.
What weak looks like: the whole file goes in, secrets included. Or, at the end, they cannot tell you which lines they wrote and which arrived.
The tell: ask at the close, what would you not ship from this, and why? A blank answer is an answer.
A task that surfaces it: put something sensitive in the environment on purpose. A config with a real-looking key, a log with personal data. Do not mention it. See who notices.
Four rows, three levels each, same for every candidate:
| Behavior | Below bar | At bar | Above bar |
|---|---|---|---|
| Choosing | Reflex, same for every task | Has rules of thumb, can name a case where they would not | Chooses per task, notices when steering costs more than doing it |
| Context | Rewords the same request | Supplies error and constraints up front | First answer lands close, adjusts once |
| Checking | Ships what compiles | Runs it against the real case | Verifies functions exist, walks the edges, reads error paths |
| Ownership | Pastes everything, cannot separate their work from the output | Follows the rule, can explain what they kept | Spots the sensitive data unprompted, names what they would not ship |
Fill this in from the recording, not from memory. Interviewers who score from memory score the person who talked most confidently. That is the calibration problem we wrote about in the bar-raiser problem.
Being honest about the limits, because a rubric that overclaims gets ignored.
You cannot get any of this from a quiz, and you cannot get it from an editor with a run button either, because most of these behaviors happen outside the editor: in the terminal, in the logs, in what the person chose to open.
EasyEnv gives the candidate a real Linux machine with the stack, the codebase and the fault already in place, then records the whole session, terminal and screen. So when you fill in the table above, you are reading what happened, not what someone remembers. Written answers get an AI-graded first pass so the reviewer starts from evidence.
Two honest limits. Someone has to review the session, which costs real time even with recordings you can skim. And a task like this needs authoring: the planted defect, the sensitive value in the config, the boilerplate that is genuinely boring. That is an afternoon of work the first time, and it is why most teams never build one.
"I use AI daily" is not information. The four things that are: whether they choose or reflex, whether they give the model facts or adjectives, whether they run the code before trusting it, and whether they know what not to send.
Pick one task this quarter, allow the tools, and watch the first five minutes. You will not need the rest of the hour to see the difference.
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
The phrase went into the job ad and nobody can say what a candidate has to show to meet it. Here is AI literacy broken into four graded behaviors with level descriptors you can screen, interview and train against.
Four assessment vendors price the same thing four different ways, so the quotes are not comparable. Here is how to read each model, and the three costs that never appear on any of them.
Read more
Most JavaScript coding tests measure whether a candidate can recall array methods, not whether they can work in a real codebase. Here are five tasks that test the second thing, plus the rubric to score them.
Read more
