AI Security Literacy: How to Test What Candidates Paste Into a Prompt

Everyone assessing AI use is assessing the output. Does the code work, is it idiomatic, did the candidate notice the model was wrong. All worth knowing. None of it touches the risk that actually shows up in a postmortem, which is not what the model produced. It is what somebody gave it.

A leaked key does not appear in the diff. A customer record pasted into a chat window to "help it understand the schema" leaves no trace in the pull request. Your code review is looking at the answer, and the exposure happened in the question.

This post is about the half of AI literacy that nobody grades: what a person is willing to paste, what they check before they paste it, and how to see that in an interview rather than in an incident channel.

What AI security literacy actually covers

It is not "knows that prompt injection exists". It is four habits, and each one is visible in how somebody works.

Knowing what leaves the building. Whether a given tool sends context to a third party, whether that context is retained, and whether the answer changes for a self-hosted model versus a hosted API. An engineer who has never asked this question about their own editor is not being careless on purpose. They have simply never been told the question exists.

Sanitising before, not after. The instinct to replace real values with representative fake ones before asking, rather than pasting production output and hoping. This is a habit you can watch somebody have or not have, in about ninety seconds.

Reading generated code for what it reaches for. Models suggest dependencies. Some of those packages do not exist, some exist and are not what the name suggests, and some quietly widen what your service can talk to. The skill is noticing an unfamiliar import and checking it rather than running it.

Knowing when the model is the wrong tool entirely. Some problems should not be described to anything outside your network, however good the answer would be. Recognising one of those is a judgment call, and it is the one that separates people who have thought about this from people who have read a policy.

Why your existing checks miss all of it

Code review sees the artifact. Every one of these failures happens upstream of the artifact.

Secret scanning catches a key committed to the repository. It does not catch the same key pasted into a prompt an hour earlier, because that never touched your repository. By the time scanning has an opinion, the value has been somewhere else for an hour.

Policy training measures whether somebody can recognise the right answer in a multiple-choice question. It does not measure what they do at 2am with a stack trace they cannot reproduce and a deadline. Those are different faculties, and only one of them is load-bearing.

And a take-home cannot see it either. The candidate does the pasting on their own machine, off camera, and hands you the result. Whatever they fed the model to get there is exactly the information you wanted and exactly the information a take-home structurally cannot capture.

Four exercises that surface it

These work in a live session on a real machine, because all four depend on watching somebody work rather than reading what they produced. In EasyEnv the candidate has a real box, an AI assistant, and the session is recorded, so the sequence of what they did is reviewable afterwards rather than remembered.

1. The stack trace with something in it. Hand them a failing service and a log that includes a real-looking connection string, an internal hostname, and a customer email. Ask them to work out why it is failing, with the assistant available. Watch what goes into the prompt. The strong candidate redacts as they go, or pastes the exception and not the surrounding lines. The weak one selects the whole log because selecting the whole log is easier, and never notices what was in it.

Nothing in this exercise mentions security. That is deliberate: the moment you say the word, everybody performs. You are looking for what somebody does when they think they are being assessed on debugging.

2. The unfamiliar import. Give them a task where the assistant is likely to suggest a package they do not know. Score whether they check it before they run it, and how. Reading the source, checking the download count and publish date, and asking why a formatting task needs network access are all good answers. Installing it and moving on is the finding.

3. The self-hosted question. Mid-session, ask which parts of what they just did they would be comfortable doing against a customer's production data, and what would have to change. You are not scoring the policy they recite. You are scoring whether they have a mental model of where the data went at all, which becomes obvious within a sentence or two.

4. The refusal. Include one subtask where the honest answer is "I would not use the model for this". Something involving credentials, or a system whose behaviour you cannot verify from outside. A candidate who runs everything through the assistant because that is what the interview seemed to want has told you something about how they will behave when nobody is watching either.

Scoring it without turning it into a quiz

Three bands, and you can place someone reliably from one session.

Unaware. Pastes whole logs, whole files, whole config. Installs suggested packages without looking. Cannot say where the context went when asked directly. This is not a character flaw and it is often fixable, but you need to know it before the offer rather than after.

Rule-following. Redacts because they have been told to redact. Correct behaviour, no model underneath it, which shows the moment they meet a case the rule did not anticipate. Ask a follow-up about a slightly different tool and watch the answer come apart.

Modelled. Talks about the data path unprompted. Redacts differently depending on the tool. Can say which of today's tasks they would have done differently against a real customer, and why. This is the person you want anywhere near production, and there are fewer of them than the job market implies.

What this is not

It is not a reason to ban the tools. A team that forbids AI assistance gets the same pasting into personal accounts on personal laptops, with none of the visibility. The exposure does not go away, it moves somewhere you cannot see it.

It is also not the most important thing in a technical interview. Whether somebody can do the job comes first. This is a second axis, cheap to add to a session you are already running, and it catches a class of failure that no amount of code review will.

The reason to assess it at hiring time is simply that it is expensive to fix later and nearly free to see now. You are already watching them work for forty-five minutes. The only change is putting something in the log that should not be pasted, and paying attention to whether it gets pasted.

Where to start

Take an interview you already run. Add one real-looking secret to the fixture data, keep everything else identical, and review the recordings for a month. You will learn more about your candidate pool from that one change than from adding a security round.

If you want the wider frame first, AI Literacy vs. AI Fluency covers what separates using the tools from understanding them, and "AI Literate" Is in the Job Description. Now Someone Has to Grade It is about turning that into something you can actually score.

What would your last five hires have pasted into that log?