What a deepfake detector is being asked to do
A deepfake detection system is a binary classifier over media: given an image, an audio clip or a video it returns a score, and a threshold turns the score into a verdict. The text detector benchmark protocol carries over whole: the score means nothing without the population it was tested on, the threshold is the tester's to fix, and accuracy without the base rate flatters the larger class. ISO/IEC TR 29119-11 calls the missing oracle the central difficulty.
Media adds the manipulation family: face swap, where an identity is replaced; reenactment, where expressions are transferred onto a real face; full synthesis, where a generator produces the whole frame; and audio-driven lip-sync, where the mouth is re-rendered to a new voice track. A detector trained on one family often fails on another, as the FaceForensics++ benchmark showed across four families and compression levels.
Building a held-out, labelled evaluation set
The real half is media whose origin the team can vouch for, captured on devices the team controls or drawn from sources with a documented chain of custody, dated after the vendor's training cut-off. The manipulated half is produced or collected with the method and generator recorded per item. Both halves must match production in resolution, codec, compression and lighting. Public benchmark sets are the first thing a vendor trains on: assemble your own and keep it private. Record source, date, family, tool, codec, compression and SHA-256 per item, then screen for near-duplicates.
Each manipulation family is its own equivalence class. A recall figure is valid only for the family it was measured on, and under 50 items the interval is too wide to act on.
Fixing the pass criteria before the run
Three numbers are written down and time-stamped before any file is scored. The false positive ceiling is the share of real media the detector may flag; it matters most because the real class dwarfs the manipulated class in production, and why base rates decide what a flag means applies unchanged to pixels. The recall floor per family is the share of each family the detector must catch. The interval is what the set size supports: with 400 real items, a 3.5 percent false positive rate has a 95 percent Wilson interval of roughly 2.1 to 5.8 percent, and the ceiling is judged against the upper bound.
The threshold is fixed at the same time, the vendor's default if one ships, and never moved after seeing results; sweeping thresholds afterwards turns a test into a marketing document. The ceiling is the call a manager makes when ranking what to test by the cost of being wrong: the false positive that names an innocent person is the expensive failure.
Testing deepfake detection: the protocol
Assemble and label the set
At least 400 real and 400 manipulated items, four or more families of 100, matched for resolution, codec and compression. Record source, date, family, tool and SHA-256 per item.
Write the criteria
State the false positive ceiling, the recall floor per family, the interval method and the threshold, and time-stamp the document.
Score every item once
Submit each file as stored, without resizing or re-encoding, through the production interface. Capture the raw score, the verdict, the model version and the time.
Build the confusion matrix by family
Tabulate the four cells overall and per family, compute the false positive rate and recall with intervals, and write pass or fail per criterion.
Check Content Credentials
Run every file through a pinned C2PA validator and record the outcome: valid and trusted, valid but untrusted, invalid, or no credentials. Cross-tabulate against the verdict.
Record the run
Store the label file, scores, validator output, model and validator versions, trust list version and criteria, read-only and hashed.
Worked example: 400 real and 400 manipulated items
A platform trust team ran the protocol on an image and video detector with 400 real and 400 manipulated items in four families of 100. The counts are invented but realistic.
| Manipulation family | Recall (invented) | What the family exercises | Typical weakness |
|---|---|---|---|
| Face swap | 94 of 100 | Identity replacement; seams at the face boundary | Heavy compression after the swap |
| Reenactment | 89 of 100 | Expression transfer; temporal consistency | Single-frame scoring misses motion artefacts |
| Full synthesis | 92 of 100 | Whole frame from a generator; frequency traces | Traces vanish after resizing or a second encode |
| Audio-driven lip-sync | 77 of 100 | Mouth re-rendered to a new voice; audio alignment | Rare in face-swap training data |
Against the criteria the run fails twice. The false positive ceiling was 3 percent and the interval's upper bound of 5.8 sits above it, so the ceiling cannot be shown to hold. The recall floor of 85 percent was missed by lip-sync at 77 of 100, whose interval of 67.8 to 84.2 never reaches it. As a single accuracy of 92.3 percent, both failures disappear.
The provenance half: what a C2PA manifest adds or subtracts
The classifier asks whether the bytes look manipulated. Content Credentials answer another: did a named signer produce these bytes, and what actions did it assert? Under C2PA 2.2 the four outcomes a credentials validator returns combine with the verdict. A trusted camera manifest whose hard binding matches the file is evidence of capture that a low score merely agrees with; a valid manifest whose actions list a generative fill is a disclosure that makes the verdict almost redundant. An invalid manifest means the bytes changed after signing, nothing about why. No credentials, the commonest state, adds nothing.
Cross-tabulate the two. In the worked example, 61 of the 400 real items carried trusted camera manifests and 3 of those were flagged: a false positive on a signed capture is the case to route to a human first. Of the manipulated items, 22 carried manifests asserting synthetic or edited content and the detector missed 4: provenance rescuing a classifier. And 9 carried invalid manifests, because a tool edited pixels without re-signing: a pipeline defect, not a detector finding. NIST AI 100-4 calls detection and provenance complementary controls.
Pitfalls that produce flattering numbers
- Compression mismatch. A detector tested on lightly compressed files and deployed on platform re-encodes loses recall it never had. Include production compression levels.
- Resizing before scoring. An upload path that downscales the file before the model sees it removes the traces the model relies on. Score the bytes as stored, then the bytes production submits.
- Near-duplicate leakage. Frames from one clip, or an original and its swapped twin, count one item many times. Group by clip and identity.
- One generator only. A manipulated half built with one tool measures that tool. Use at least two generators per family.
- Accuracy without the base rate. In production the real class dominates and a 3.5 percent false positive rate can outnumber every true detection. Report the two rates separately.
How to report it
Report the set first: counts per class and family, sources, dates, generators, codecs, compression levels and the near-duplicate screening. Then the criteria as written before the run. Then the confusion matrix overall and per family, every rate as a count over a denominator with a 95 percent interval, plus the threshold and model version. Then the manifest state per item and the cross-tabulation, with validator name, version and trust list. Then pass or fail per criterion, noting that recall holds only for the families tested. The fields a stranger needs to re-run it are in what an evidence-grade record contains; the hub for testing AI systems and authenticity pipelines covers the interval arithmetic; and the family table is the equivalence class method applied to media.
Common questions
How many items does a deepfake detection test need?
Enough to bound the rates you care about. With 400 real items a 3.5 percent false positive rate carries an interval of roughly 2.1 to 5.8 percent; with 100 per family, recall intervals span about 15 points. Under 50 cannot pass or fail a floor.
Why report recall per manipulation family instead of overall?
Because detectors generalise poorly across families. A detector trained on face swaps can catch 94 of 100 swaps and 77 of 100 lip-sync edits, and an overall 88 percent hides the family that will fail in production.
Should the files be resized before they are scored?
No. Resizing and re-encoding remove the traces a detector relies on, so score the bytes exactly as stored. If production downscales before scoring, run a second pass on the downscaled bytes and report both; the second number is what users experience.
What does a valid C2PA manifest add to a detector verdict?
A trusted camera manifest whose hash matches the file is evidence of capture that a low score agrees with. A manifest whose actions assert generative edits is a disclosure. An invalid manifest means the bytes changed after signing, nothing more; no credentials adds nothing.
Is accuracy a useful headline for a deepfake detector?
Not on its own. On a balanced set accuracy averages two rates the reader needs separately, and in production the real class dominates, so the false positive rate decides how many flags are wrong. Report both rates with denominators and intervals.
Testing deepfake detection is discipline rather than tooling: a held-out labelled set with families recorded, criteria fixed before the run, one threshold, rates with denominators and intervals, manifest state against verdict, and a record a stranger can re-run. A detector that passes has been measured; one that ships with a single accuracy figure has been advertised.
Sources
- NIST AI 100-4, Reducing Risks Posed by Synthetic Content: An Overview of Technical Approaches to Digital Content Transparency (2024)
- C2PA Technical Specification 2.2: manifests, content bindings, validation and trust lists
- Rössler et al., FaceForensics++: Learning to Detect Manipulated Facial Images (ICCV 2019), a benchmark of detectors across four facial manipulation families and compression levels
- ISO/IEC TR 29119-11:2020, Guidelines on the testing of AI-based systems
Further reading named in the text
- Rössler, Cozzolino, Verdoliva, Riess, Thies and Nießner, FaceForensics++: Learning to Detect Manipulated Facial Images, ICCV 2019
This guide is part of the testing AI systems hub. It is best read alongside how to benchmark an ai text detector and false positives, base rates and the confusion matrix, which cover the neighbouring questions.