How we test AI tools

The protocol every test follows, the rubric behind every score, what can change a score, and a public log of every dispute.

MethodUpdated 21 October 2026

In short

Every tool sees the same frozen, fingerprinted sample of 1,500 documents at one fixed setting. We publish each rate by slice with its 95 percent interval, score it with one published, weighted rubric in which false flags on human writing count most, give each maker its numbers before publication, and log every rerun and dispute.

Run this test on your own sample

The benchmarking guide turns this protocol into steps you can follow.

Open the benchmarking guide

The protocol

Freeze the sample
Documents are collected, labelled and fingerprinted before any tool sees them.
Fix the settings
Plan, interface, version and the flag rule are written down before the first document.
Run and record
Every document runs once per tool, in a fixed order; every raw reply is kept.
Score by slice
Rates with 95 percent intervals per slice, then the weighted rubric.
Reply, then publish
Each maker sees its own numbers 7 days before the report goes live with its date.

A test answers two questions: how often a tool wrongly flags writing by people, and how often it catches AI text in the forms it really arrives in. Both are measured on the same sample, under rules fixed before the first document is run. The full protocol, the data format and the checker the test team uses are published with every report.

The sample

Each round uses 1,500 documents of 300 to 800 words, all on the same list of 60 topics so that topic never explains a difference between human and AI results. Human documents are written in supervised sessions or come from writers who consented to their use; none is scraped. AI documents are generated by four current models from four makers and recorded with the model name, version and date.

Tier A sample: every tool sees the same 1,500 documents
Slice	What it is	Documents
Native human writing	Essays, reports and letters written by native English writers, with provenance and consent	300
Second-language human writing	The same kinds of writing by writers whose first language is not English	300
Unedited AI text	Generated by four current models from four makers, 75 documents each, on the same topics	300
Lightly edited AI text	The unedited AI documents with 10 to 30 percent of sentences revised by a person	300
Paraphrased AI text	The unedited AI documents run through a paraphrasing tool in its default mode	300

When a document counts as flagged

A document counts as flagged when the tool's own verdict says AI-written or mixed, or when its AI score is 50 or more on a 0 to 100 scale. The rule is the same for every tool and is written into each run before it starts. Tools that only give a label are judged on the label; tools that only give a number are judged on the number.

Scoring rubric

The overall score is the weighted sum of five parts, out of 100. False flags on human writing carry the most weight because a wrong flag can lead to a false accusation, which is the harm readers of this site most often face.

The five parts of every score, with their weights
Rubric	Weight	What it measures	How it is scored
False flags on human writing	35%	Share of all human documents the tool flags as AI, native and second-language writers together	100 at 0 percent, minus 10 points per percentage point, never below 0
Catching AI text	25%	Share of unedited and lightly edited AI documents the tool flags	The detection rate as a score: 80 percent flagged scores 80
Holding up after paraphrasing	15%	Detection on paraphrased AI text, compared with detection on the same text unedited	Paraphrased rate as a share of the unedited rate, capped at 100
Fairness to second-language writers	15%	Gap between the false flag rate on second-language writing and on native writing	100 with no gap, minus 10 points per percentage point of gap, never below 0
Explaining its result	10%	Whether the tool shows why it scored as it did	Five checks, 20 points each: a number, marked passages, a published false positive rate, a stated model version, a warning against sole-evidence use

Every rate has a 95 percent interval (Wilson method). We recompute the score at the best and worst ends of those intervals and publish the range. Two tools whose ranges overlap are shown as tied, whatever their order in the table.

A tool is labelled lower false flag risk only if the upper end of its interval is 2 percent or less on both human slices. No tool is ever labelled safe for deciding misconduct cases: no detector score should be the only evidence in one.

What we publish

Counts, rates and 95 percent intervals for every tool on every slice, with the tool version and test date.
The flag rule, the rubric version and the score, with the range the score could take within those intervals.
Rerun agreement: the share of documents that got the same decision when run again on a different day.
Per-document scores for the AI documents, and fingerprints for every document so the sample cannot be changed after the fact.
Never published: the human documents themselves, which are used with their writers' consent for testing only.

What changes a score

Changes a score

A rerun on a new tool version, logged with its date
A corrected label in the sample, applied to every tool
An upheld dispute after a rerun

Never changes a score

A tool maker's request without a rerun
Payment, referral terms or advertising of any kind
A rerun we did not publish in the test log

Right of reply

Seven days before a report is published, each tool maker receives its own numbers, the method and the protocol version. A reply received in that window is summarised in the report. After publication, makers and readers can dispute a result by email to the editor with the subject line Dispute. We answer within 14 days, rerun the test when the claim can be tested, and log the outcome below whether or not it changes anything.

Disputes log

No disputes have been logged yet: the first reports have not been published. Each entry will show the date, the tool, the claim, what we reran and the outcome.

Protocol version

Protocol v1, published 21 October 2026. A change to the sample design, the flag rule or the rubric gets a new version number and is listed here with its date; earlier results keep the version they were run under and are not rescored silently.