August 17, 2026·5 min read
You hired someone with a beautiful take-home. Clean notebook, sensible feature work, a genuinely good score on the held-out set. Everyone was impressed.
Eight months later there is still nothing in production. The model works on their machine. It has never been evaluated against the metric the business cares about. Nobody knows what it costs to serve, and when the input data shifted in March, nothing noticed.
This is the most common ML mis-hire, and it is not a talent problem. You tested the one part of the job that was never the bottleneck.
Modeling is the visible part, the part that is taught, and the part with tidy public benchmarks. So that is what interviews test.
But in most companies the model is rarely the reason a project fails. The reasons are: the training data quietly disagreed with the production data, the offline metric had nothing to do with the business outcome, serving was too slow or too expensive to be worth it, and nobody built a way to notice when it degraded.
An engineer who is a seven out of ten at modeling and strong on those four will ship. Someone who is a ten and weak on them will produce excellent notebooks forever.
So test the four.
The most valuable ML instinct is suspicion about the input.
The task. Give them a dataset and a model that scores suspiciously well. Plant one real problem: a feature that leaks the label, a train/test split done before deduplication, or a target computed with information that will not exist at prediction time. Ask them to decide whether to trust the result.
Strong: they doubt the number before they admire it. They check how the split was made, look for duplicates, and inspect the top features for anything that could not be known at prediction time. They find the leak, and they can explain why it would look fine offline and fail in production.
Weak: they accept the score and start tuning hyperparameters.
This one discriminates harder than anything else on the list, and it takes twenty minutes.
Nearly every failed ML project we hear about had a good offline metric and no connection to the outcome anyone wanted.
The task. Give a business situation with an unequal cost of error. Fraud, where a missed case costs a hundred times a false alarm. Or triage, where the model's output feeds a queue with limited human capacity. Ask them to define how they would evaluate it, and what threshold they would ship at.
Strong: they ask what a false positive costs and who handles it. They reach for a metric that reflects the asymmetry, they think about the operating point rather than the whole curve, and they talk about how they would check it after launch, not just before.
Weak: accuracy. Or a metric with no argument attached to why it is the right one.
The task. Give them a trained model and a small serving endpoint that is too slow. Real environment, real requests, real latency numbers. Ask them to get p99 under a target.
Strong: they measure before changing anything. They find where the time actually goes, which is usually preprocessing or loading rather than inference. They know the levers: batching, a smaller model, caching, moving work out of the request path, and they can say what each one costs in accuracy or complexity.
Weak: they optimize what they assume is slow. Or they reach immediately for more hardware without knowing where the time went.
Models fail quietly. That is what makes this the skill teams regret not testing.
The task. Show them a model that has been live for six months and a set of monitoring dashboards. Something has drifted. Ask what is wrong and what they would have put in place to catch it sooner.
Strong: they look at input distributions, not just output accuracy, because labels usually arrive late or never. They think about what signal is available without ground truth. They mention retraining triggers and how they would avoid retraining on data the model itself polluted.
Weak: "we would monitor accuracy." Ask where the labels come from and watch what happens.
Not a separate exercise so much as a question to attach to the others: what does this cost to run per thousand predictions, and is that worth it?
Seniority shows up here faster than anywhere else. Engineers who have operated a model in production have a number, or at least a way to get one. Engineers who have not have never been asked.
If you are hiring for LLM work rather than classical ML, the weights shift. Modeling matters less, since the model is usually somebody else's. Three things matter more:
Evaluation design becomes the main skill. Anyone can build a demo that works on five examples. The job is building an eval set that catches regressions when a prompt changes or a provider ships a new version. Ask them how they would know a change made things worse. A candidate without a real answer here will ship a system that degrades invisibly.
Failure handling. What happens on a timeout, a refusal, a malformed response, a rate limit. This is ordinary engineering and it is where most LLM prototypes fall over on the way to production.
Cost and latency awareness. Token spend per request, what gets cached, when a smaller model is enough. Same question as above, sharper teeth.
Prompt-writing skill is real but small, and it changes with every model release. Do not build the interview around it. The related reading here is everyone says they use AI daily on assessing AI use generally.
Every task above needs something running. A slow endpoint is not a slow endpoint in a text box. A drifted model needs six months of data behind it. A leaky dataset needs to be explorable.
EasyEnv gives the candidate a real Linux machine, or several networked together, with the dataset, the model, the serving endpoint and the monitoring already in place. Every candidate gets an identical setup built from the same recipe, so the comparison is fair, and the environment is destroyed afterwards. The session is recorded, terminal and screen, so you can see whether they measured before they optimized, which is the single most informative moment in the latency task.
Honest limits. These tasks take real effort to author, more than a coding question does, because a plausible dataset and a realistic failure are work to build. And they need a reviewer who understands ML; AI-graded first passes help with the written reasoning but they will not make the final call for you. If you only need to check that someone knows what regularization is, this is heavier than you need.
Test the parts that decide whether a model reaches production: suspicion about data, evaluation that matches the business, serving under real constraints, and noticing failure without labels. Modeling ability will be visible through all four anyway.
If you change one thing, make it the first task. Hand a candidate a model that scores too well and see whether they get suspicious or get excited. That reaction tells you most of what you need to know.
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Timers, proctoring, no-docs rules and think-aloud requirements filter for things you never meant to test. Here is what each one actually measures, the replacement that keeps the difficulty, and how to offer accommodations without a negotiation.
At high volume the real constraint is engineer attention, not candidate quality. Here is what automation genuinely handles at each stage, what it quietly makes worse, and how to find out whether your funnel is throwing away good people.
Read more
Recruiters are asked to make a technical call they were never equipped to make, then blamed for the shortlist. Here are questions that need no technical knowledge to evaluate, and the one artifact worth handing to the hiring manager.
Read more
