Do AI detectors work? What the published tests found
AI text detectors work in a narrow sense: on clean machine output and fluent native-speaker writing, most of them sort the two better than chance. Outside that case the published evidence is consistent and unflattering. This page collects what independent researchers measured, with the numbers, so that a teacher, editor or writer can see what a detector result can and cannot carry.
Foundation guide6 min readUpdated
Run this test on your own sample
The protocol guide sets out the sample set, the threshold and the report in the order you need them.
Open the protocolThe short answer
No published independent test has found an AI text detector accurate enough to be the only evidence that someone did not write their own work. Detectors can be useful as a prompt to look closer. They fail most often in exactly the situations that matter: writers whose English is a second language, text that has been edited or paraphrased, and short passages.
What four studies measured
Where detectors fail most
Second-language writers. Liang and colleagues found that the same seven detectors were near-perfect on US 8th-grade essays and wrong on most of the non-native essays. Their explanation is that simpler, more predictable word choice looks statistically like machine text. When the researchers had a language model enrich the vocabulary of the non-native essays, the average false positive rate fell from 61.22 percent to 11.77 percent. The guide to fairness testing for non-native writers shows how to check this for the detector you use.
Edited and paraphrased text. Weber-Wulff and colleagues report that no tool in their test classified all edited or paraphrased AI documents correctly. Liang and colleagues found that a single follow-up prompt asking a chatbot to rewrite its essay in more literary language cut detection from up to 100 percent to up to 13 percent. The method for measuring this is in adversarial testing of AI text detectors.
Short texts and other languages. The same developer said its classifier was very unreliable below 1,000 characters, performed significantly worse outside English, and should not be used as a primary decision-making tool.
Why a small error rate still means many wrong flags
A detector that wrongly flags 1 percent of human writing sounds safe. Run it on 1,000 honest essays and it flags about 10 of them. If only a few essays in the class were actually machine-written, a large share of all flags can be wrong. The working is in false positives and base rates, which includes a calculator you can run with your own numbers.
Is there a 100 percent accurate AI detector?
No. None of the published independent tests found one, and the developers' own documentation describes limits on short text, other languages and edited text. A claim of 99 or 100 percent accuracy usually describes one test set of clean machine output and fluent human writing. Ask for the sample size, the false positive rate on human text, the date and the version, and how the tool did on edited and non-native writing. Those four facts are what a detector benchmark should report.
What a detector result can be used for
- A reason to look more closely at a piece of work, alongside other signs such as a sudden change in style or knowledge the writer cannot explain.
- A starting point for a conversation with the writer, not a conclusion.
- Never the only evidence for a penalty, and never on its own for a writer whose first language is not English.
If you have been flagged, the strongest answer is evidence of process: version history, drafts, notes and sources. See proving you wrote it. If two detectors gave you different answers, why AI detectors disagree explains why that is normal.
Many universities have reached the same conclusion and switched their detector off; see what AI detectors universities use for 17 published policies.
What Testing Experience is adding
Published studies age as tools change. We are running our own dated tests of current detectors and humanizers on a fixed, documented sample set, with false positive rates reported separately for native, non-native and short writing. Results will appear in the AI Tool Tests section with the tool version, test date and full method, and this page will link to them. The method, the scoring rubric and the right of reply are published in advance on how we test.
This guide is part of the testing AI systems hub. It is best read alongside false positives, base rates and the confusion matrix and what an ai detector score means for a writer, teacher or editor, which cover the neighbouring questions.
Questions
Do AI detectors actually work?
Only in a limited sense. Independent tests find they sort clean machine text from fluent native writing better than chance, but none of the 14 tested in a 2023 study reached 80 percent accuracy across realistic documents, and they fail often on non-native and edited writing.
Is there a 100 percent accurate AI detector?
No. No independent test has found one. Claims of 99 or 100 percent usually come from a single clean test set and leave out false positives on human writing.
Is there an AI detector that actually works for teachers?
None is reliable enough to decide a case. A detector result can justify a closer look or a conversation, but not a penalty on its own.
Are free AI detectors less accurate than paid ones?
Price is not a guide to accuracy. The 2023 study tested 12 publicly available tools and two commercial systems, and all 14 scored below 80 percent accuracy.
Why do AI detectors flag non-native English writers?
Detectors look for predictable word choice, and writers using a second language often choose simpler, more predictable words. In one study, seven detectors flagged non-native essays as AI-written 61 percent of the time on average.
Can paraphrasing beat an AI detector?
Often, yes. Researchers found paraphrasing dropped one detector's catch rate from 70.3 percent to 4.6 percent. This is also why a low score does not prove a text is human-written.
Why was a major chatbot developer's AI classifier withdrawn?
The company stated on 20 July 2023 that the classifier was no longer available due to its low rate of accuracy. In its own evaluation it caught 26 percent of AI-written text and wrongly flagged 9 percent of human text.
What should I do if a detector flagged my work?
Collect evidence of how you wrote it: version history, dated drafts, notes and sources, and offer to explain the work. Point to the published false positive rates, especially if English is not your first language.
Sources
- Weber-Wulff et al., Testing of detection tools for AI-generated text, International Journal for Educational Integrity 19, 26 (2023)
- Liang et al. (2023), detectors of machine-written text are biased against non-native English writers
- Krishna et al., Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- New AI classifier for indicating AI-written text, classifier announcement (2023, withdrawn 20 July 2023)
- Debora Weber-Wulff and colleagues, Testing of Detection Tools for AI-Generated Text (International Journal for Educational Integrity 19, article 26, 2023)
- New AI Classifier for Indicating AI-Written Text, announcement by its developer (31 January 2023, updated 20 July 2023)
