03
Why AI detectors fail, and who pays for it
Tools that promise to tell you whether a text or an image was machine-made are unreliable in a specific, predictable direction — and the people they wrongly accuse are rarely the ones who can argue back.
7 min read
There is a large market in software that claims to detect machine-generated content. Schools buy it. Employers buy it. It does not work well enough for the decisions people make with it.
The structural problem
A detector is a classifier trained to separate two populations. It works by finding statistical regularities in generated output that human output lacks.
Three things break it.
The target moves. Every new model shifts the distribution the detector was trained on. Detection is permanently behind, by construction.
The output is adversarial. Anyone who wants to evade a detector can paraphrase, prompt for a different register, or run the text through another model. The countermeasure is trivial and widely known.
The populations overlap. Plenty of human writing is regular, hedged and evenly textured — technical documentation, legal drafting, anything written by someone taught to write formally. It sits exactly where the detector expects generated text.
The direction of the error
This is the part that matters ethically, and it is not symmetric.
Detectors have been shown to flag writing by non-native speakers at substantially higher rates. Someone writing in a second language often uses a smaller vocabulary, more conventional constructions and steadier sentence lengths — the same features a detector reads as machine-like.
So the error lands on students and staff writing in a language that is not their first, and on anyone whose prose is plain by training or by discipline.
A test whose false positives concentrate on one group is not a slightly imperfect test. It is a mechanism for producing a particular kind of accusation.
The confidence score problem
Most of these tools return a percentage. “87% likely AI-generated.”
That number reads as a probability about the document. It is nothing of the kind — it is an output of a model whose calibration on your specific text is unknown, and it carries no error bar.
But it is a number, and numbers get treated as evidence in a way that a shrug does not. An institution that would never expel a student on a hunch will do so on a percentage.
What to do instead
For text, in an educational or professional setting, the honest answers are procedural rather than forensic:
- Ask the person to talk through their argument. Someone who wrote it can.
- Look at drafts and version history, which are far harder to fabricate convincingly than a finished document.
- Set work that requires specific, local, verifiable material.
- Change what is being assessed, rather than trying to police the tool.
For images and video, the answer is provenance, which is the subject of the next piece.
If you are the one accused
Ask for the tool’s false-positive rate on writing by non-native speakers. Ask what the score means numerically. Ask whether the institution has validated it on its own student population.
Most cannot answer any of the three, and the request tends to change the conversation.