October 28, 2026·5 min read
Ask any team whether they have backups and the answer is yes. Ask when somebody last restored one and the room goes quiet.
That gap is the most reliable source of bad days in infrastructure. The job that writes the dump has been green for two years. Nobody has ever read the dump back, and so nobody knows that it has been truncated since a disk filled up in March, or that it contains the data but not the schema, or that restoring it takes nine hours on a database that the business expects back in one.
It is also one of the easiest things to interview for, because the task is unambiguous and the environment is cheap: one box, one database, one dump from last night.
An outage story is rehearsed by the second interview. Everybody has one, everybody tells it well, and it tells you about their narration rather than their hands.
A restore is the opposite. There is a working system or there is not, there is a number for how much data was lost or there is a shrug, and none of it can be prepared in advance, because the specific mess is yours.
The setup. A box with an application that expects a database, the database gone or corrupted, and a dump from last night sitting on disk.
Ask: get us back to working.
Strong looks like: they look at the dump before trusting it: size, format, whether it contains what it should. They restore into a scratch database first and check it, rather than straight over whatever is in place. They bring the application up afterwards and confirm it actually works rather than announcing the restore finished.
Weak looks like: restoring directly over the live target as a first move. It usually works. When it does not, there is now neither a working database nor an untouched broken one to investigate.
Why it works: this is the task, and most candidates have genuinely never done it end to end on a system they did not build.
The setup. The same restore, with traffic that was still arriving after the dump was taken, visible in application logs.
Ask: how much data is missing, and which?
Strong looks like: they answer with a time window and a rough count rather than a feeling. They know the dump's timestamp, they find the last write it contains, and they look at the logs to say what happened after it. Then they say what to do about it: replay from a queue, re-import from a partner, or tell support exactly which customers to contact.
Weak looks like: "a few hours, probably". During a real incident, somebody is about to tell customers something, and the difference between "about four hours" and "from 02:14 to 06:40, around nine hundred orders" is the difference between a believable company and a nervous one.
Why it works: recovery point is the number the business asks for first, and most engineers have never had to produce it under time pressure.
The setup. A second backup file that is present, recent, and broken. Truncated mid-write, or missing the schema, or dumped with a tool version that cannot be read by the one installed.
Ask: use this one instead.
Strong looks like: they detect the problem quickly rather than waiting for a nine minute restore to fail at the end, and they can say what they would add so this never passes silently again: a restore test on a schedule, a checksum, an alert on size deviation. The best candidates point out that a backup nobody has restored is not a backup, which is the thesis of the whole exercise.
Weak looks like: repeated attempts with different flags, with no hypothesis about why the file is bad.
Why it works: silent backup failure is extremely common, and the fix is a process change rather than a command, which makes it a seniority question.
The setup. A larger dataset and a stated expectation: the business wants service back within an hour. The naive restore takes longer than that.
Ask: we have an hour. What do you do?
Strong looks like: they treat it as a prioritisation problem, not a speed problem. Restore what the application needs to serve traffic first, bring the rest in behind it, or stand up a degraded read-only mode while the full restore runs. They state the trade honestly, including what stays broken meanwhile, and they tell you what they would change afterwards so the restore is faster next time.
Weak looks like: running the same restore faster, or promising an hour without having measured anything.
Why it works: recovery time is a business constraint, and the candidates who have lived through this talk about it in terms of what the company can do without, which is exactly the instinct you want. The behavioural side of the same hour is covered in the live incident drill.
Four rows, the same for every candidate:
None of those rows is "knows the backup tool". Any competent engineer learns the flags in an hour. The habits above take an incident to acquire, which is exactly why you would rather they arrived with them.
This needs a real box, a real database, real dumps including a deliberately broken one, and permission to destroy all of it. Nobody can run this on staging, and a candidate cannot be given your production backups, for obvious reasons.
EasyEnv gives each candidate a disposable box with the database, the application, last night's dump and the broken one already in place, built from the same recipe every time, so two candidates face the identical situation and the comparison is fair. They can wipe the database, because it is theirs and it is gone afterwards. The session is recorded, so you can see whether they inspected the dump or just ran the command and hoped. The same recipe doubles as an internal drill for the team you already have.
The limits, plainly. Preparing the scenario takes a day, including making a backup that is broken in a realistic way rather than obviously. The review needs somebody who has done a real restore. And for a role with no production responsibility at all, this hour is better spent elsewhere.
Everyone has backups. Ask the candidate to restore one, on a box they have never seen, with a dump that may or may not be good.
If you only run one, ask for the number. An engineer who can tell you within minutes how much data was lost, and which customers it belonged to, has done this before, and that is the person you want on the call.
Run live coding sessions and take-home challenges in real production environments. Watch sessions back, score consistently, and hire with confidence.
More posts you might like
Everyone says they know Git until a bad merge lands on main and the team loses a morning. Here are four tasks on a real repository, a conflict that is not textual, a commit that has to come out, a branch that has drifted, and a bug with an unknown origin.
Most API interviews turn into a style argument about plurals and status codes. The decisions that cost money are elsewhere. Here are four tasks on a running service that test retries, compatibility, pagination and error design, with what a strong answer sounds like.
Read more
Substitution is rare, real, and concentrated at the volume end of hiring where nobody meets anybody before the offer. ID uploads and webcam checks mostly inconvenience honest candidates. Here is what actually makes a stand-in impossible, and what it costs.
Read more
