Somebody added "comfortable working with AI tools" to the job description. Now you are running the round that checks it, and your own experience is a few months of asking a chat window to draft things.
This is not a competence problem. It is a sequencing problem: the requirement arrived before the expertise did, and it landed on hiring managers rather than on the people who had time to develop a view.
The risk is specific. An interviewer who cannot evaluate a domain well tends to score confidence, and in this domain confidence is negatively correlated with the thing you want. The engineers with the strongest instincts talk about tradeoffs and uncertainty. The ones who tell you AI has tripled their output are the ones shipping code nobody has read.
Not how to build with models. Enough to tell a real answer from a fluent one.
What the tool does and does not see. A model answering about a codebase is answering from what was put in front of it. Almost every candidate failure and success traces back to what context they supplied, and if you do not know that, you will credit the model's answer to the candidate.
That confident and correct are unrelated. Models produce the same tone whether they are right or inventing a library that does not exist. Knowing this changes what you listen for: not whether the candidate got an answer, but whether they checked it and how.
That "I use it for everything" is a finding, not a qualification. The people who use it well use it selectively and can tell you where it stops paying. Somebody who cannot name a task they would not use it for has not thought about it. AI Literacy vs. AI Fluency is the longer version of this distinction.
That the risk sits in the input. Reviewing generated code catches bad code. It does not catch the production log with credentials in it that somebody pasted to get the answer. Your biggest AI risk is not bad code covers what that means for the round.
That speed is not the outcome. Faster to a first draft is easy and nearly universal. Faster to something correct is the actual claim, and it depends entirely on how good the person is at rejecting what they were given.
That is the floor. It takes an afternoon of reading and one honest hour using the tools on something real.
You do not need to out-technical the candidate. You need questions where a shallow answer is visible to you.
1. "Show me something it got wrong recently, and what you did." The best question in this round. Everybody has an example, and the answer separates instantly. A strong candidate has a specific one: a hallucinated method, an API that changed, a subtly wrong concurrency suggestion, and they can say how they caught it. A weak one says it is usually pretty good, which means either they do not check or they are not doing hard enough work.
2. "What do you not use it for?" Looking for a real boundary with a reason. Anything involving credentials, code they will be on call for, decisions where being 90% right is worse than not answering. The non-answer is "I use it for everything, you just have to check the output".
3. "How do you check it, specifically?" Push past "I review the code". Running it, writing the test first, reading the library's actual source, checking that a package exists before installing it. Vague verification is the most common gap and it is the expensive one.
4. "What did you paste in?" Asked about a story they just told you. This is the security question in a form that does not announce itself, and the answers vary enormously.
None of these require you to evaluate code. All four are answerable by a hiring manager listening for specificity.
They tell you what somebody believes about their own practice, which is not the same as their practice. Everybody now knows the correct-sounding answers, because they are in every article on this subject including this one.
The only thing that closes that gap is watching somebody work. In EasyEnv the candidate has a real machine, a real broken system and an AI assistant available, and the session is recorded. The interesting part is not the code at the end. It is the sequence: what they asked, what they accepted, what they threw away, and the moment the model was confidently wrong and either they caught it or they did not.
That recording is also the thing that fixes your own literacy fastest. Watch five candidates work with a model on the same problem and you will have a calibrated sense of what good looks like that no amount of reading provides. You will also see how differently the strong and weak ones behave in the first ninety seconds, which is the part nobody describes accurately in an interview answer.
Do not ban the tools to sidestep the question. An interview that forbids AI measures a working style that no longer exists on your team, and the candidate knows it.
Do not let one enthusiastic engineer own the round. The most vocal AI user on your team is not automatically the best judge of AI judgment, and a round designed by one person's preferences becomes a test of whether the candidate works the way that person does.
You can run this round well without being an expert, on one condition: be clear with yourself about what you are measuring. You are measuring whether somebody has a real, examined practice, whether they verify, and whether they know where the tool stops. You are not measuring how impressive their AI stories sound, and the difference between those two is where most of these rounds go wrong.
"AI Literate" Is in the Job Description. Now Someone Has to Grade It has the rubric, and everyone says they use AI daily covers the four skills worth testing separately.
When did a model last tell you something confidently wrong, and how did you catch it?
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
The first infrastructure hire is not a scaled-down platform engineer. Here is what the role really is at that stage, why tool-list interviews mislead, and four scenarios that show whether someone can work inside a system nobody documented.
Time to hire is mostly waiting, not interviewing. Here is where the days actually go, which ones you can remove without lowering the bar, and the two changes that cut the most.
Read more
The Go language is easy to read and hard to run in production. Here is what a Go developer assessment should actually test, four scenarios that separate people who have shipped Go from people who have written it, and why a syntax-shaped test tells you nothing.
Read more
