October 14, 2026·6 min read
Everyone applying for an AI role can write a good prompt. That stopped being a differentiator some time ago, and the demo always works, because demos are run by the person who built them on the input they had in mind.
The job is what happens afterwards. The same feature, four months later, called by people who type strangely, against a model that was updated last week, inside a budget somebody has to defend.
That is an engineering problem, and it is testable the same way other engineering problems are: on a running service that is already slightly broken.
One thing to be clear about first. This is not the same interview as assessing an engineer who uses a model to write code. We cover that separately in AI agent skills assessment and four AI skills and how to test each. This post is the other seat: the person building the feature the model sits inside.
The setup. A service that asks a model for structured output and then uses it. Roughly one call in ten comes back in a shape the code does not expect: an extra sentence before the JSON, a field missing, a number as a string. The current code throws, and the user sees an error.
Ask: this fails for about one user in ten. Make it reliable.
Strong looks like: they validate the output against a schema at the boundary instead of trusting it, and they decide deliberately what happens on a bad response: retry once with the error fed back, fall back to a narrower request, or fail honestly with a message that makes sense. They treat the model as an unreliable external service, which is exactly what it is, and they log the failures so the rate is visible rather than anecdotal.
Weak looks like: a longer prompt that asks more firmly for JSON. It raises the success rate, proves nothing, and leaves the failure path exactly as it was.
Why it works: it separates people who have shipped a model-backed feature from people who have demoed one. The demo never showed them the tenth call.
The setup. A prompt somebody wants to improve, a handful of real inputs, and no evaluation of any kind. The candidate is asked to make a change and show that it helped.
Ask: make this better, and convince me it is better.
Strong looks like: they build the smallest useful eval before they change anything: a set of inputs with expected properties, run before and after, with results they can show. They are honest about variance, and they know a sample of five does not settle anything. The best candidates ask what "better" means for this feature before measuring, because precision and recall pull in opposite directions depending on who is reading the output.
Weak looks like: a rewritten prompt and a confident claim. It might be better. Nobody will ever know, including them.
Why it works: this is the single highest-value habit in the role. Teams with an eval harness improve steadily; teams without one change prompts forever and argue about vibes.
The setup. A conversational feature that appends everything to the context. Long sessions get slower and more expensive, and eventually hit the window and fail in a way that looks random to the user.
Ask: here is the cost per conversation over the last month. Explain the shape, then fix it.
Strong looks like: they connect the cost curve to the growing context immediately. They talk about what can be dropped, summarised or stored outside the prompt, and they know that the fix changes behaviour, so they say what quality cost they are accepting. They also put a cap in place, because an unbounded context is a billing incident waiting for a patient user.
Weak looks like: a smaller model. It reduces the number without addressing the shape, and the failure at the window comes back later anyway.
Why it works: cost and context management is where AI engineering stops resembling ordinary backend work, and it is where a junior-level answer is most visible.
The setup. The upstream model API starts returning rate limit errors and occasional long latencies. The service retries immediately, three times, and has a 60 second timeout, so a page that used to render in two seconds now hangs for three minutes before failing.
Ask: the model API is degraded. What should our users experience?
Strong looks like: they answer the product question first: what does the user see when this feature cannot work right now. Then the engineering, backoff with jitter, a cap, a circuit breaker, a queue for work that does not have to be synchronous, and a timeout chosen from how long a person will actually wait. They mention which calls are safe to retry, because retrying a request that already charged money is a different conversation.
Weak looks like: more retries, or no retries. Both are positions taken without reference to what the user is doing while they wait.
Why it works: every model provider has bad afternoons. How a service behaves during one is a product decision that engineers make by default, usually without noticing.
Four rows, the same for every candidate:
None of those rows is "writes good prompts". That is visible throughout, and it is the easiest part to learn.
These tasks need a service that is really calling a model, a cost report, a provider that can be made to misbehave and a session long enough to blow a context window. None of that fits in a document.
EasyEnv gives the candidate a box with the service running against a stubbed provider we control, so we can return malformed output on demand, inject rate limits and slow responses, and do it identically for every candidate. That matters more here than in most interviews: a live provider makes the test unrepeatable and the comparison unfair. The box is disposable and the session is recorded, so you can see whether they built an eval or just edited the prompt.
The limits, plainly. Building the stubbed provider is a day of work the first time. Reviewing needs somebody who has shipped a model-backed feature, which is a small group in most companies and worth borrowing from another team. And if the role is really ML research rather than applied engineering, this is the wrong interview: see the machine learning engineer assessment instead.
Stop grading prompts. Give the candidate output that does not parse, a change they have to prove, a context window that is costing real money, and a provider having a bad afternoon.
If you only run one, run the eval task. An engineer who insists on measuring before claiming an improvement will keep your AI features working long after the model underneath them changes.
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
You cannot ship an app in a 60 minute interview, so mobile hiring falls back on algorithm puzzles and a portfolio walkthrough. Here are four tasks inside a real project, a leak that survives the screen, state lost on rotation, a slow network, and a crash only on resume.
Give an analyst clean tables and a clear question and nearly everyone looks capable. Here are four tasks on a deliberately messy database, duplicated rows, nulls that mean two different things, a timezone that moves a month boundary, and a number that disagrees with finance.
Read more
PHP hiring still runs on framework questions while the failures are operational and silent. Here are four tasks on a running Laravel application, a queue nobody noticed had stopped, a listing page that issues hundreds of queries, a migration that ran on one server, and uploads that vanish behind a load balancer.
Read more
