You give the candidate a task. They describe it to the assistant. Nine seconds later there is a complete, plausible, well-formatted implementation on the screen.
Everything you were going to assess just happened without you. Can they write a loop, do they know the standard library, is the code tidy: answered, by something that is not the candidate.
The interview has not ended, though. It has moved. Everything worth knowing about this person is in what they do next, and most interview formats are not built to see it.
Do they read it. The first fork, and it is more discriminating than it should be. Some candidates read the generated code line by line before running it. Some run it, see it work, and move on. Under interview pressure, with something that appears to work, the pull towards moving on is strong, and it is exactly the behaviour that produces a codebase nobody understands.
Do they notice the thing that is subtly wrong. Generated code is usually right and occasionally wrong in a specific way: an edge case unhandled, an error swallowed, a query that works on ten rows and not ten million, a library call whose semantics changed two versions ago. Whether somebody catches that depends on whether they have a model of the problem independent of the model's answer.
Can they steer it. When the first output is not right, weak candidates rephrase the same request more emphatically. Strong ones change the problem: give it the constraint it was missing, narrow the scope, supply the error, or decide to write this part themselves and use the model for the next part. Steering is a skill and it is visible within two exchanges.
Do they know when to stop asking. Every session has a point where the model is going in circles and the candidate should reason directly. Recognising it fast is one of the most valuable habits in AI-assisted work, and it separates people more cleanly than almost anything else.
The design constraint is simple: if the assistant can complete the task alone, you learn nothing. That rules out most conventional exercises, since a model will implement a rate limiter, a parser, or a CRUD endpoint perfectly.
What models cannot do alone is anything that depends on the specific system in front of them.
A bug in a running system. The model can suggest ten causes of a connection timeout. It cannot know which one is happening on this box without the candidate going and finding out. The assistant becomes a hypothesis generator, and the candidate's job is elimination, which is the actual skill.
A change in an unfamiliar codebase with real conventions. The model will write correct code in the wrong idiom. Whether the candidate notices and adapts it is the assessment.
Requirements that are incomplete on purpose. Leave out something a model will confidently assume. The strong candidate asks. The weak one ships the assumption, and now you know something.
Something that must actually run. Any task where the answer is checkable by execution removes the "sounds right" failure entirely.
Notice these are the same properties that make a good interview task without AI. The tools did not change what is worth assessing. They removed the exercises that were only ever proxies.
This needs a real environment and a recording. Not a screen share where you watch someone type, which is stressful and where you miss half of what happens, and definitely not a take-home, where all of the interesting behaviour occurs off camera.
In EasyEnv the candidate gets a real box with the broken system, the assistant available and the session recorded. Afterwards you can review the sequence: what they asked, what came back, what they kept, what they threw away and what they went and checked themselves. That sequence is the assessment. The final diff, which is the only thing a take-home gives you, is the least informative artifact in the session.
Say clearly at the start that AI is allowed and expected. A candidate who is unsure whether using it counts against them will behave strangely for the first ten minutes, and you will have spent your best ten minutes measuring their anxiety.
The pattern is consistent enough to describe.
They read the task and ask one clarifying question before touching anything. They look at the system first, not the assistant, long enough to have their own idea of what is happening. When they do ask, the prompt contains specifics: the actual error, the actual constraint, the relevant part of the file. They read what comes back. They reject part of it out loud, keep part of it, and adapt the rest to the codebase's conventions. At some point they say the model is not getting it and reason directly, and they are right that it was not getting it. They run things constantly. When it works, they can explain why it works without referring to the assistant.
The weak session is also consistent: the whole task described in one prompt, the answer pasted in whole, an error, the error pasted back, another answer pasted in, repeat. It sometimes converges. When you ask why the final version works, the answer is a paraphrase of the model's explanation.
Both of these are legible in a recording and hard to fake, because they are pacing and order rather than content.
Run this for a quarter and you will find candidates who look weaker than their CV, and some who look considerably stronger. The second group is usually people whose independent judgment is good, and the model amplifies it. The first group is often people whose previous work was carried by the parts a model now does.
That is not a reason to avoid the format. It is the format telling you what you hired for last time.
How to design an interview that reveals AI judgment covers the task design in more depth, and how to run a pair-programming interview covers the human version of the same session.
In your last AI-assisted interview, could you say what the candidate contributed?
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Hiring managers are now expected to judge how well candidates work with AI, mostly without having done it themselves on real work. Here is the minimum literacy you need to run that round honestly, and the four questions that expose whether you have it.
The first infrastructure hire is not a scaled-down platform engineer. Here is what the role really is at that stage, why tool-list interviews mislead, and four scenarios that show whether someone can work inside a system nobody documented.
Read more
Time to hire is mostly waiting, not interviewing. Here is where the days actually go, which ones you can remove without lowering the bar, and the two changes that cut the most.
Read more
