September 15, 2026·6 min read
Backend HiringHands On AssessmentPython Developer AssessmentRole Specific AssessmentTechnical Interviews
Ask a QA candidate to define the test pyramid and you will learn that they have read about the test pyramid. Ask them what they would delete from your suite and you find out whether they have owned one.
QA is the discipline where the interview drifts furthest from the work. Testing is an adversarial, evidence-gathering job, and it gets assessed with definitions. This post is about assessing it the way it is done: on a running application, with a real suite, in front of failures that are genuinely ambiguous.
Three reasons, all understandable.
Testing skill is hard to see quickly. A developer candidate produces code you can read. A tester's output is a judgment about risk, which looks like nothing until it turns out to be right or wrong months later.
The obvious questions have known answers. Boundary values, equivalence partitioning, the pyramid, black box versus white box. All real, all recall, all on the first page of any testing blog.
The automation half gets tested as programming. So the loop becomes a weak developer interview, and you select for people who write test code rather than people who decide what is worth testing. Those overlap less than teams expect.
The result is a loop that a strong manual tester fails and a mediocre Selenium author passes.
If you take one thing from this post: put a genuinely flaky test in a real suite and hand it over.
Not a puzzle test that fails for a cute reason. A test that passes about eight times in ten because of a shared fixture, a timing assumption, or an ordering dependence. The kind your own team has quietly learned to re-run.
Why it works so well:
It is the real job. Every team above a certain size has these, and how they are handled is the difference between a suite people trust and a suite people ignore.
There is no single right answer, so the reasoning is exposed. Quarantine it, fix it, delete it, or leave it and file the bug: all defensible, depending on what the test protects.
It separates two habits cleanly. Strong candidates reproduce the failure before touching it: run it in a loop, run the suite in a different order, run it in isolation, look for state that survives between tests. Weak candidates add a sleep or a retry and declare it fixed. Both answers arrive in twenty minutes, and only one of them tells you what your Friday deploys will look like.
It is a values question in disguise. Ask what they would do if the fix takes a day and the release is tomorrow. You learn how they trade off confidence against velocity, which is most of the job.
The regression they cannot reproduce. A bug report from a user with vague steps, a running app, and access to logs. What you watch: whether they narrow. Good candidates form a hypothesis about which layer, find the one signal that would rule it out, and move. They also write down the exact steps once they have them, because a bug that cannot be reproduced cannot be closed.
The feature with no requirements. Show a working feature and say: here is what it does, tell me what you would test. What you watch: coverage of the unglamorous. Empty states, permissions, double submit, the back button, what happens when the network drops halfway. Strong candidates ask who uses this and what happens if it silently does the wrong thing, because that question determines how much testing it deserves.
The suite that takes fifty minutes. A test suite too slow to run on every change. What you watch: whether they reach for parallelism, or for deletion. The best answer usually involves both, plus an argument about which tests actually protect anything. Anyone who proposes running the full suite nightly and calling it solved has not felt the cost of finding a break twelve hours after it landed.
Every scenario above needs software that runs. A flaky test is not flaky in a document, and a regression cannot be narrowed without logs.
In EasyEnv the candidate gets a real machine with the application, its dependencies and its test suite, pre-broken in the way you chose. The session is recorded, so the review is about sequence: did they reproduce before fixing, did they check for shared state, did they re-run enough times to know the fix held. In testing, sequence is nearly the whole signal, because the discipline is precisely about not believing something until you have evidence.
It also settles the AI question honestly. Let them use a model. It will write a plausible test plan for any feature you name. It cannot tell them which of two hundred existing tests is the one that fails intermittently on your CI runner, and watching a candidate go and find that out is the assessment. The same logic as designing take home challenges that survive AI applies here.
Score four things, not the fix:
Reproduction. Did they make the failure happen on purpose before changing anything?
Isolation. Did they narrow to a cause, or apply a general remedy and hope?
Risk judgment. Can they say what a test protects and what it would cost to lose it?
Communication. Could a developer act on their report without asking a follow-up question? Write out a bug report at the end of the session and read it as if you were on call. This is the part that manual testers are often outstanding at and automation-heavy loops never measure.
Put those four on a sheet before you start. How to write a technical interview rubric covers the mechanics.
Do not ask for a test plan for a hypothetical app. You get a generic document, from the candidate or from their model, and it is indistinguishable between a good tester and an average one.
Do not treat automation code style as the main axis. Page objects and fixtures matter and they are learnable in weeks. Deciding what is worth automating is not.
Do not run the loop without a developer in the room for at least one session. Half of QA quality is how the tester works with engineers, and a panel of testers cannot see that.
What is the flakiest test in your suite right now, and what has it actually caught?
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Almost every Python candidate can write a comprehension and a decorator. Far fewer can say why the worker grew to 6GB overnight. Here are the tasks that separate them, run on a real box with a real process.
A breakdown of the actual question formats you will see on a HackerRank test, what each one is really checking, and how to spend your prep time before the timer starts.
Read more
A live system design interview lives or dies on structure, not on which question you pick. Here is a minute-by-minute run sheet for keeping it a conversation, plus when a take-home design doc is actually the better call.
Read more
