What a benchmark has to establish
A detector benchmark answers three questions for a specific population: how often does it flag text a person wrote (the false positive rate), how often does it catch machine text in the form it will actually arrive (the detection rate, after editing and paraphrase), and do those rates hold on every group of writers it will be used on. Everything in the protocol below exists to make those three numbers trustworthy. A single accuracy figure answers none of them, because it mixes the two error types and depends on the ratio of human to machine text in the test set, as the guide to false positives and base rates shows.
The published record is a reason for caution, not for cynicism. Weber-Wulff and colleagues tested fourteen detection tools in 2023 and found none of them both accurate and reliable, with performance falling further on edited and obfuscated text. Liang and colleagues, publishing in Patterns the same year, found that several detectors flagged more than half of a set of essays by non-native English writers as machine-written, while essays by native writers were classified far more accurately. Krishna and colleagues showed that paraphrasing machine text sharply reduced detection by several published detectors. None of this says a detector is useless; it says the vendor's number is not your number until you have measured it.
The protocol
Define the population and the decision
Write down who produces the documents the detector will judge (for example, first-year university essays in English, a third of them by writers with another first language) and what happens to a flagged document. The decision sets which error matters most.
Build the human-written corpus
Collect documents known to be human-written, from before generative tools were common or produced under supervision, matched to the population on genre, length and writer background. At least 300 per slice you intend to report.
Build the machine-written corpus
Generate documents on the same prompts and topics with several current models and settings. Add slices for the forms real submissions take: lightly edited by a person, paraphrased by a tool, and mixed documents with machine paragraphs inside human text.
Quarantine the corpus
Keep the evaluation documents out of any tuning, threshold search or demo. If the detector's supplier has seen the documents, the result measures memory, not detection. The test set design guide covers leakage in detail.
Fix the threshold in advance
Choose the score above which a document counts as flagged before running the benchmark, ideally the threshold that will be used in production. Tuning it on the test set inflates every result.
Run and record every score
Score every document, record the raw score as well as the verdict, and keep the run reproducible: detector version, settings, date, corpus version.
Compute the matrix and the intervals
Build the confusion matrix per slice. Report the false positive rate and the detection rate with Wilson 95 percent intervals, not bare percentages.
Report against the real base rate
Convert the rates into the precision you should expect at your population's plausible share of machine text, and state it next to the rates.
Slices that must be reported separately
| Slice | Why it matters | What to watch |
|---|---|---|
| Writers with another first language | Simpler, more regular prose scores as more machine-like on many detectors | False positive rate against native writers in the same genre |
| Short documents | Less text means a noisier score | False positive rate below about 250 words |
| Formulaic genres | Lab reports, legal summaries and abstracts are predictable by design | False positive rate per genre |
| Edited machine text | The realistic case: a person revises a draft | Detection rate drop against unedited text |
| Paraphrased machine text | The adversarial case | Detection rate at the fixed threshold |
| Mixed documents | A few machine paragraphs in human work | Whether the tool localises or averages |
Reading the numbers
In the worked result above the detector flagged 20 of 1,000 human documents. The point estimate is a 2.0 percent false positive rate; the Wilson 95 percent interval is 1.3 to 3.1 percent, which is the range a careful reader should carry away. With only 100 human documents and 2 false flags, the same 2 percent estimate would have an interval of 0.6 to 7.0 percent, too wide to support any decision. That is why the protocol asks for hundreds of human documents per slice. When a slice has no false positives at all, the rule of three gives a quick one-sided 95 percent upper bound of 3 divided by the sample size: zero false flags in 300 documents bounds the rate below about 1 percent, not at zero.
A detector result is a score from one instrument, with a measured error rate. It is a reason to look closer, never evidence on its own. A benchmark report should say so on its first page.
Writing the report
A useful benchmark report fits on two pages. Page one: the population, the decision, the threshold, and a table of false positive rate and detection rate per slice with intervals. Page two: the confusion matrices, the corpus description (sources, dates, sizes, how machine text was produced), the detector version and settings, and the limitations. Treat it like any other test metric: never show the detection rate without the false positive rate beside it. The companion guide on the three-layer approach to testing AI systems places this benchmark inside a wider test approach.
Common questions
What is the most important number in an AI detector benchmark?
The false positive rate on human-written text from your own population, per subgroup, with a confidence interval. It measures how often a person is wrongly flagged, which is the error with the highest cost in most uses.
How many documents do I need to benchmark a detector?
At least 300 human-written documents per slice you want to report, and a similar number of machine-written ones. Fewer gives confidence intervals too wide to act on; zero false positives in 300 still only bounds the rate below about 1 percent.
Why do independent benchmarks report lower accuracy than vendors?
Different test sets. Independent studies use edited, paraphrased and non-native text and fix the threshold in advance; vendor figures often come from clean machine text and native writing. Both can be honest and still disagree.
Should the threshold be tuned on the benchmark?
No. Tune it on a separate development set, fix it, then run the benchmark once. Tuning on the test set produces a result that will not hold in production.
Can a detector result be used as proof?
No. A benchmark gives a measured error rate, and even a low rate produces many wrong flags when most documents are human-written. A flag is a reason for a human review with other evidence, not a finding.
Sources
- Weber-Wulff et al., Testing of Detection Tools for AI-Generated Text (2023)
- NIST/SEMATECH e-Handbook of Statistical Methods, 7.2.4.1 Confidence intervals for a proportion (Wilson method)
- Liang et al. (2023), detectors of machine-written text are biased against non-native English writers
Further reading named in the text
- Kalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting and Mohit Iyyer, Paraphrasing Evades Detectors of AI-Generated Text, but Retrieval Is an Effective Defense (NeurIPS, 2023)
- Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang and Soheil Feizi, Can AI-Generated Text Be Reliably Detected? (arXiv preprint, 2023)
- Edwin B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference (Journal of the American Statistical Association 22, 1927)
- Tom Fawcett, An Introduction to ROC Analysis (Pattern Recognition Letters 27, 2006)
This guide is part of the testing AI systems hub. It is best read alongside false positives, base rates and the confusion matrix and evaluation test set design for ai models, which cover the neighbouring questions.