How a statistical watermark works
In the scheme Kirchenbauer and colleagues described in 2023, each time the model picks the next word, the preceding word is used with a secret key to split the vocabulary into a green list and a red list. The generator adds a small bonus to green words, so its output uses them more often than chance. A person reading the text notices nothing. A detector with the key recomputes the lists for every word and counts green words. In unwatermarked text about half are green; in watermarked text many more are, and a standard significance test turns the count into a z-score.
The appeal for testers is that the detection step is a well-defined statistical test with a false positive rate that can be set in advance, unlike the learned scores discussed in how AI text detectors work. With a high enough z threshold, the chance that ordinary human writing trips it is tiny.
What a watermark test can and cannot show
| Question | Can the watermark answer it? | Why |
|---|---|---|
| Did this long passage come from the watermarking system, unedited? | Often, with strong evidence | Many words carry the signal |
| Did a person use any machine at all? | No | Other generators leave no watermark |
| Is this short reply machine-written? | Rarely | Too few words for a confident test |
| Was this text paraphrased from watermarked output? | Sometimes | Paraphrasing removes much of the signal |
| Did the writer intend to deceive? | No | A statistical pattern says nothing about intent |
Removal and robustness
Because the signal lives in word choice, changing the words erodes it. Kirchenbauer and colleagues' follow-up study on reliability found that watermarks remained detectable after human rewriting and machine paraphrasing if enough text was observed, while noting that paraphrasing weakens the signal and more text is needed to recover confidence. Work on adversarial testing of detectors shows the same tension for every detection approach: the more the text is reworked, the less any fingerprint survives.
Treat a strong watermark result on a long, unedited passage as meaningful evidence about where that passage came from. Treat the absence of a watermark as meaning almost nothing.
Testing a watermark scheme
- Measure the false positive rate on a large, varied set of human writing at the chosen z threshold, including formulaic and non-native text.
- Measure detection by length band, because the evidence depends on how many words are available.
- Measure detection after light editing, paraphrasing and translation, holding the threshold fixed, and report the drop.
- Check text quality with and without the watermark, since a strong bias towards green words can make writing worse.
A worked run, with invented but realistic counts: 2,000 human-written documents of 300 to 900 words scored at a z bar of 4 produced 1 flag, a false positive rate of 0.05 percent with an upper bound near 0.3 percent. At a bar of 3 the same set produced 6 flags, 0.3 percent, and the one document flagged at both bars was a legal template full of repeated fixed phrases. Detection on 2,000 unedited watermarked documents was 1,960 at z of 4 and fell to 1,140 after a paraphrase pass, a drop of 41 points. Both halves of that record belong in the report, along with the version of the paraphraser.
Worked example: green-list counts behind the z-scores
The z-scores in the figure come from a simple calculation, shown here with invented but realistic counts. Suppose the key splits the vocabulary in half, so an unwatermarked text lands on the green list about 50 percent of the time, and the watermarking generator lands on it about 74 percent of the time. For a passage of 400 scored tokens the expected green count by chance is 200 with a standard deviation of 10, so 298 green tokens gives z of 9.8. The same passage paraphrased by a second tool keeps 232 green tokens, z of 3.2, below the bar of 4. An honest human passage of the same length lands on 206, z of 0.6.
| Text | Scored tokens | Green tokens (expected by chance) | z-score | Call at z of 4 |
|---|---|---|---|---|
| Watermarked, unedited | 400 | 298 (200) | 9.8 | Watermark found |
| Watermarked, 50-token excerpt | 50 | 37 (25) | 3.5 | Inconclusive |
| Watermarked, then paraphrased | 400 | 232 (200) | 3.2 | Inconclusive |
| Human-written | 400 | 206 (200) | 0.6 | No watermark |
| Machine text from another system | 400 | 197 (200) | -0.3 | No watermark |
Common mistakes with watermark tests
- Reading no watermark as human-written. Text from any system without this key, and paraphrased text from this one, both return no watermark.
- Lowering the z bar to catch more edited text. Every point off the bar raises the false positive rate on honest writing by a calculable amount; measure it before moving it.
- Scoring excerpts. The evidence grows with length, and a 50-token excerpt of a strongly watermarked text is inconclusive by design; score the whole text or report the token count with the result.
- Assuming the key is stable. Providers rotate keys and schemes; a test that worked last year may not read this year's output, so record which key version each run used.
- Skipping the human false positive measurement because the theory gives a rate. Real writing has repeated phrases, lists and quoted text that the independence assumption behind the z-score does not model.
Watermarks and content credentials
Watermarks for text are related to, but different from, provenance records for images and media such as the C2PA content credentials specification, which attach signed metadata rather than hiding a pattern in the content. Both answer where something came from; neither answers who is responsible for how it was used, which is why the guide to proving you wrote it focuses on process evidence. The two also fail differently. A credential is either valid or broken, and any edit outside a signing tool breaks it, so testing one is a matter of checking signatures and manifests, as the guide to verifying content credentials describes. A watermark degrades gradually with editing, so testing one is statistical. A pipeline that carries both, for example a generator that watermarks its text and a publishing tool that signs the file, needs both kinds of test, and the guide to testing content credentials pipelines covers the signing half.
| Approach | Where the evidence lives | Survives editing? | False positive control | What absence means |
|---|---|---|---|---|
| Statistical watermark | In the word choices of the text | Partly; degrades with edits | Set by the z bar, then measured | Nothing |
| Content credential | In a signed manifest attached to the file | No, unless re-signed by a tool | Cryptographic; a forged one fails | Nothing, or stripped on upload |
| Learned detector score | Inferred from the finished text | Weakly | Measured on a benchmark, often high | Weak evidence either way |
Common questions
What is a text watermark for AI?
A hidden statistical pattern that a text generator deliberately embeds in its word choices, which a detector holding the key can test for later.
Can watermarks prove a text was written by AI?
They can give strong evidence that a long, unedited passage came from the watermarking system. They cannot show whether other generators were used or what a writer intended.
Can a watermark be removed?
Editing and paraphrasing weaken it because the signal is in word choice. Research found it can survive moderate rewriting if enough text remains, but heavy rewriting removes most of it.
Do watermarks cause false positives?
Rarely at a high threshold, because the test's false positive rate can be set in advance. It must still be measured on real human writing.
Why can't short texts be checked reliably?
The test counts words that match the secret list; with only a few dozen words, chance variation is too large for a confident result.
Do text watermarks survive translation?
Mostly not. The signal lives in the specific words the generator chose, and translation replaces all of them. Published work found some signal survives light rewriting; a full translation is far beyond that.
Sources
This guide is part of the testing AI systems hub. It is best read alongside how ai text detectors work and adversarial testing of ai text detectors, which cover the neighbouring questions.