August 13, 2026·5 min read
Two thousand applications. Three engineers who will help, badly, between sprints. One role, or twelve, depending on the week.
The instinct is to look for a better filter. But the honest constraint is not candidate quality and it is not the filter. It is how many hours of engineer attention exist, and that number is fixed before you start.
Everything below follows from that. Design the funnel around the attention you have, not around the applications you got.
Work out your budget in hours.
Say three engineers can each give four hours a week for six weeks. Seventy-two hours. Now decide how you want to spend them.
Neither of those is the answer. The point is that the answer is arithmetic, not ambition. Once you have the number, every stage has to justify the hours it eats.
Two thousand applications and seventy-two hours means the first several stages must cost roughly zero engineer time. That is not a compromise. It is the shape of the problem.
Going stage by stage, cheapest first.
Deduplication and completeness. Free, do it, no downside. Same person applying to six roles, missing fields, broken links.
Hard knockouts. Right to work, location if the role truly requires it, salary band if you publish one. Fine to automate, with one rule: knockouts must be things that are genuinely binary. "Five years of experience" is not binary. Automating it is how you reject the best person in the pile.
Structured application questions. Three or four short written answers, same for everyone. Cheap for the candidate, cheap for you, and much more informative than a CV. Ask about work they have done, not about opinions.
An auto-graded floor test. A short, automatically scored task to remove people who cannot program at all. This is the one place where an auto-scored puzzle earns its keep, and it works if you use it as a floor rather than a ranking. Set the bar low. You are removing the bottom, not finding the top.
Scheduling. Self-serve booking. This is not a small thing. Scheduling round trips are usually the largest source of delay in a funnel, and automating them costs nothing in quality.
Everything up to here is close to free per candidate. Then it changes.
Work samples. Real signal, real cost. A human has to look. This is where your seventy-two hours start draining, and it is the first stage worth spending them on.
Interviews. The most expensive thing you own. Never automate the decision, only the logistics.
Three things degrade in ways that do not show up in any dashboard.
Ranking by score. An auto-graded test is a rough instrument. Using it as a gate is reasonable. Sorting candidates by it and taking the top forty is not, because differences of a few points are noise, and you have just turned noise into a shortlist. Gate, then look.
Every rejection being silent. At volume it is tempting to skip the reject email. Do not. It costs nothing and you are building the pool you will hire from next year.
Longer tests, because the tool allows it. A ninety-minute test at the top of a funnel does not filter for skill. It filters for people who have ninety minutes to spare, which correlates with having no job and no caring responsibilities and not much else. If a stage is early and unpaid, keep it under thirty minutes.
Here is the uncomfortable part. Everything above optimizes rejection, and nobody measures whether the rejections were right. You never find out about the person you filtered out.
So sample them. Once a quarter, take twenty candidates the automated stages rejected, and have an engineer spend an hour looking at what they submitted. Not to reverse the decisions. To find out whether the filter is doing what you think.
Teams that do this usually find one specific rule doing most of the damage. A knockout on a keyword. A floor test that punishes people who use a language it grades badly. A time limit that quietly excludes anyone applying after a full working day.
You will not find those any other way, because your dashboard only shows the people who got through.
For two thousand applicants and seventy-two hours:
The move that matters most is number 4. Because the work sample gives real evidence, you can run one interview stage instead of three, which is where the hours come back. A weak first stage forces you into extra rounds to resolve the uncertainty it created.
Related: how to scale technical hiring from 5 to 50 engineers covers the org side of this, and why your best candidates drop out of the funnel covers what the friction costs you.
At the top of a funnel we are not the tool. Two thousand applications need a cheap automated floor, and an auto-scored screen does that job properly. We would rather say so than sell you the wrong thing.
Where we change the arithmetic is stage 4. A candidate gets a real Linux machine with the stack and a real task: a service that will not start, a ticket in an actual codebase. Every candidate gets an identical machine, provisioned from the same recipe and destroyed afterwards, so the comparison between them is fair.
Then the part that matters for your seventy-two hours: the session is recorded, terminal and screen, and it is skimmable. Reviewing what someone actually did takes minutes, not the length of the session. Written answers get an AI-graded first pass, so the engineer opens a summary with evidence attached rather than a blank submission.
That is the trade. You spend more per candidate at stage 4 and buy back a whole interview round, plus a much smaller group reaching it.
Your constraint is engineer hours, and it is fixed. Automate everything that is genuinely binary, use an auto-graded test as a floor and never as a ranking, and spend your human attention on one stage that produces real evidence.
Then do the thing nobody does: pull twenty rejected candidates next quarter and look at them. That hour tells you more about your funnel than any report your ATS can produce.
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Recruiters are asked to make a technical call they were never equipped to make, then blamed for the shortlist. Here are questions that need no technical knowledge to evaluate, and the one artifact worth handing to the hiring manager.
Generated code fails in a different pattern from human code, which is why normal review habits miss it. Here is the review order that catches the most in the least time, and the hiring task built from it.
Read more
Every candidate now claims AI fluency, so the claim carries no information. Here is what tool choice, context giving, verification and responsible use look like when you actually watch someone work, and the tasks that surface each one.
Read more
