Five reasons for disagreement
| Reason | What differs | Effect |
|---|---|---|
| Signal | Predictability statistics against learned features | Formulaic human text fools one but not the other |
| Training data | Which generators, genres and languages were in the training set | Each detector is strongest on text like its own data |
| Threshold | Where each tool draws the line | The same underlying evidence gives different verdicts |
| Scale | Probability, percentage of sentences, or a label | Numbers that look comparable are not |
| Length and mixing | How short or partly edited text is handled | Verdicts diverge most on exactly these texts |
The first reason is structural. As the guide to how AI text detectors work explains, a statistical detector and a trained classifier are looking at different properties of the text. A technical manual full of standard phrasing is predictable, so a statistical detector leans towards machine; a classifier trained mostly on essays may never have seen anything like it and lean the other way.
Scores are not on a common scale
One tool reports a probability that the whole document is machine-written. Another reports the share of sentences it flagged. A third reports a label with a confidence word. A 60 from one and a 60 from another do not mean the same thing, and neither is necessarily a calibrated probability. Calibration can be checked: take 1,000 documents the tool scored between 55 and 65 and count how many were really machine-written. If the answer is 20 percent rather than 60, the scale is a ranking and nothing more, and most published checks of this kind have found exactly that. Comparing raw scores across tools is like comparing test coverage figures computed with different coverage criteria: the numbers share a unit sign and little else.
Disagreement clusters where detection is weakest
Independent testing shows the pattern. Weber-Wulff and colleagues evaluated fourteen detection tools on human, machine, edited and translated texts and found none reliable enough to use as evidence, with manual editing and machine paraphrasing reducing detection sharply. Where every tool is uncertain, small differences in design decide the verdict, so tools disagree. Sadasivan and colleagues argued that the gap between human and machine text narrows as generators improve, which leaves every detector less room to be right and more room to disagree. Long, unedited machine output is where tools agree most, and it is also the case that matters least in practice.
If detectors disagree about a text, the honest conclusion is that detection cannot tell for this text. Running more detectors until one agrees with a suspicion amounts to choosing the answer in advance.
Worked example: one paragraph, three tools, two rounds
The figure above shows the three raw outputs. What matters is what each tool did with its number. The values are invented but realistic: a 150-word paragraph from a cover letter, scored before and after the writer spent ten minutes editing it.
| Detector | Output type and threshold | Round 1 (original) | Round 2 (after edit) | Verdicts |
|---|---|---|---|---|
| A: statistical | Probability the document is machine-written; flag above 80 | 88 | 71 | Flagged, then passed |
| B: classifier | Share of sentences flagged; flag above 50% | 34% | 22% | Passed both times |
| C: classifier, stricter | Label with a confidence band; label above 60 | 61, "likely mixed" | 48, "unclear" | Labelled, then unclear |
Round 1 alone already gives three answers. Round 2 shows why: on a text this short each score rests on a few dozen word choices, so an ordinary edit moves it by a large margin, and a threshold placed anywhere in the middle of the range will be crossed by editing alone. The paragraph did not become more human between rounds. It became less formulaic, which is a different property, and the only one any of the three tools can see. Had the writer instead pasted in a machine-written paragraph and edited it for the same ten minutes, the scores would have moved in the same direction by a similar amount.
Common mistakes when comparing detectors
- Averaging the three numbers. 88, 34 and 61 are a probability, a share of sentences and a banded label; their mean is nothing.
- Taking the strictest tool as the safest. A stricter threshold raises the false positive rate on honest writing along with the detection rate, so the strict tool is the one most likely to accuse an honest writer.
- Rerunning until a tool agrees with a suspicion, then citing that one. The run record should show every tool that was tried and what each said, in the order they were tried.
- Testing a 150-word sample when the full document is available. Scores stabilise with length; short excerpts are the least reliable input on every approach, and the one most likely to produce three different answers.
- Comparing tools on a handful of texts. The published comparisons used hundreds of documents per category and still found the tools unreliable; three texts show nothing.
What to record when tools disagree
- Tool and version
- name, version or date of the model behind it
- Input
- the exact text scored, its word count, whether it was an excerpt
- Raw output
- the number or label on the tool's own scale, before any threshold
- Threshold
- the tool's default or the value the organisation set, and who set it
- Order
- which tools were run first and why the next one was run
If detectors are run at all, record every run: the tool and version, the exact text scored and its length, the raw output on the tool's own scale, the threshold applied and the date. Three tools run on the same day with all five fields filled in are a small evaluation; the same three tools with only the verdicts written down are an argument waiting to happen. A disagreement recorded this way is useful evidence that the text sits in the uncertain zone. A single cherry-picked verdict is not. Where a detector result feeds a public label such as "AI-assisted" on a piece of content, that label has its own failure modes and needs its own test, which the guide to testing AI disclosure labels sets out: a label applied on the strength of one tool's score inherits that tool's false positive rate and adds a display layer that can also go wrong.
What to do instead
- Look at the writing process rather than the finished text: drafts, notes, version history and a conversation with the writer, as the guide to proving you wrote it describes.
- If a detector is used at all, use one, tested on your own population, at a fixed threshold, with its error rates known.
- Never combine scores from different tools into an average; the scales are not compatible.
- Read the guide to what a detector score means before acting on any single result.
Common questions
Why do different AI detectors give different results?
They use different signals, were trained on different data, apply different thresholds and report on different scales. On short, edited or mixed text these differences decide the verdict.
Which AI detector is the most accurate?
It depends on the text. Accuracy measured on one kind of writing does not transfer to another, and independent studies found none reliable enough to serve as evidence on its own.
Should I use several detectors and take the majority?
No. Their errors are not independent and their scales differ. Agreement is weak corroboration; disagreement shows the text is outside what detection can judge.
Why does the same detector give different scores after small edits?
Because the score reads statistical patterns that small edits change, especially in short texts. Instability under small edits is a sign the result is not robust.
Are detectors better on longer texts?
Generally yes. Longer texts give more words for the statistics, so scores are more stable. Very short passages are unreliable with any approach.
Is a disagreement between detectors evidence that the text is human-written?
No. It shows the text is in the region where detection is unreliable, which is also where edited or mixed machine text lands. Disagreement means detection cannot decide; it does not decide in either direction.
Sources
This guide is part of the testing AI systems hub. It is best read alongside how ai text detectors work and what an ai detector score means for a writer, teacher or editor, which cover the neighbouring questions.