September 25, 2026·6 min read
A pull request lands. It is clean. Good names, consistent style, tests included, a tidy description. You approve it.
Six weeks later something breaks in a way nobody can explain, and it turns out the fix is in a file that three people have now read and none of them really understand.
This is the new review problem. Code arrives faster than anyone can verify it, and it arrives looking finished. The old habit, reading until something feels off, does not work, because nothing feels off.
Human mistakes cluster around the hard parts. The author struggled with the tricky bit, so the tricky bit is where the bug is. Reviewers learned to read that way, and it worked.
Generated code fails in the opposite pattern. The mistakes are in the parts that look easiest: confident, ordinary, well-formatted code that happens to be wrong.
These are the ones that keep coming up:
There is an eighth that is worse than any of these: a confident explanation that is wrong. The code may be fine and the reasoning beside it teaches the reader something false. That one spreads.
The first three steps catch most of it, and they take about five minutes.
1. Check every symbol is real. Before reading for logic, jump to the definition of anything unfamiliar. Check the version you actually have installed, not the docs the model half-remembered. Ninety seconds, and it catches the most embarrassing failures.
2. Run it against the real case. Not the test that came with it. Model-written tests routinely assert the behavior the model implemented, bug included. Take one case from the actual requirement and run it.
This is the highest-value step and the one people skip, because the code looks done.
3. Say the edges out loud. Empty. One. Many. Null. Zero. Negative. Huge. Concurrent. Unicode. Crossing midnight. For each, ask whether this code does something sensible. You will find the missing one in under a minute or confirm there is not one.
4. Read the error paths first. Skip the happy path on the first pass. It is the part the model is best at. Go to the catches, the retries, the timeouts, the cleanup. What happens when the dependency is slow, down, or half-answers?
5. Ask if it belongs here. Does it use your utilities, your conventions, your logging? Correct-but-foreign is a real cost even though nothing is broken.
6. Check the security defaults. Query building, input validation at the edges, anything touching secrets or crypto.
7. Ask if you could maintain it. The last gate, and the strict one: can you explain every line to someone else? If not, you are not reviewing it. You are accepting it.
What to skip: formatting, naming style, micro-optimizations. That is what generated code is already good at. Every minute spent there is a minute not spent on step 2.
Deciding what to accept is now a distinct skill, and it varies wildly between candidates. It is also harder to fake than writing code from scratch, since writing from scratch is the part a model can do for them.
The setup. A real repository, a plausible ticket, and a pull request that was produced by a model. Tell the candidate exactly that: this was AI generated, decide what to do with it.
What to plant. Four defects, at different depths:
Give them the environment with the dependencies installed and let them use anything, including a model. You are testing whether they verify, not whether they can spot bugs by staring.
What actually separates people. From sessions we have reviewed, the split shows up early:
Scoring. Four rows: defects found, whether they ran it or only read it, the quality of the merge or reject decision, and whether they can explain the code they would keep. Use the same rows for everyone, in the shape from how to write a technical interview rubric. Do not score speed on its own.
A review task only works if the code can actually run. If the candidate cannot install the dependency, hit the endpoint, or execute the failing case, then "check it against reality" is not available to them and you are back to grading whether they read carefully.
EasyEnv gives them a real Linux machine with the repo cloned, dependencies installed and the service running before they join. So step 2 is one command away, and whether they took it is the clearest signal in the session. The terminal and screen are recorded, so in the debrief you can see the exact moment they either ran the code or decided they did not need to. Written answers get an AI-graded first pass so the reviewer starts from evidence.
The trade-off is real. Authoring a task like this takes an afternoon: you have to plant defects that fail the same way for every candidate, and someone has to review the session afterwards. There is no green checkmark. If you need a cheap automated filter over thousands of applicants, this is not that, and an auto-scored screen is the honest answer for that step.
Check the symbols exist. Run it against the real case. Name the edges out loud. Then read the error paths. Style comes last, if at all.
And refuse to merge anything you cannot explain. That single rule catches more long-term damage than the rest of the list combined, and it is the one worth putting in your review guidelines this week.
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Under the EU AI Act, AI used to screen or evaluate candidates sits in the high risk category. Here is what that actually asks of you as an employer, the questions to put to your assessment vendor, and how to structure a process you could defend.
When the model has shell access, the question is no longer whether a candidate can write the code. It is whether they can supervise something that writes it faster than they can read it. Here is how to test that on a real repository.
Read more
Observability interviews usually become tool interviews, which sort candidates by which vendor their last employer bought. Here is how to test the actual skill: narrowing down a failure that only happens to four percent of requests.
Read more
