Evidence-grade test records for authenticity systems

Minimum fields

8 groups, see the table

Inputs

SHA-256 per file, stored read-only

Criteria

fixed and time-stamped before the run

Storage

write-once, hashed manifest, signed

Reproduction

by a second tester, from the record alone

Retention

as long as the decision can be challenged

When a test record becomes evidence

Most test records are read once, by the team, deciding whether to ship. A record about an authenticity system has a second audience that arrives later and reads it adversarially: a student appeals panel quoting a false positive rate, a photographer disputing a correction, a court receiving a verification session as an exhibit. That reader wants a reason to disbelieve the result, and an ordinary test report offers several: no hashes, no versions, criteria apparently written after the numbers were known.

The test documentation standard, ISO/IEC/IEEE 29119-3, already asks for most of the right content: completion criteria, test data, environment, execution logs and incident reports. Evidence grade adds integrity and reproducibility. Those techniques come from digital forensics, where NIST SP 800-86 and RFC 3227 describe collecting and preserving evidence for incident response, and they apply almost unchanged to test artefacts.

What makes evidence-grade test records different

None of these fields is exotic. In an evidence-grade record every field is present for every run, inputs are identified by hash, and the record is sealed rather than saved. Teams that already report detector results with intervals, as in confidence intervals for classifier metrics, usually have the numbers and lack the integrity layer.

Chain of custody for test artefacts

Chain of custody is a log of who held an artefact, when, with proof that it did not change while they held it; for digital artefacts the proof is a hash. Hash every input before the run and every output immediately after, and record them in the run log. Store the artefacts in write-once storage, where objects cannot be overwritten or deleted for a retention period. Whenever an artefact is copied or handed to someone else, log the hash before and after the hand-off.

Then seal the record: write a manifest listing every artefact with its hash, hash that manifest, sign it with a key held by a named person or the organisation, and obtain a time-stamp from a time-stamp authority so that the signature can be shown to have existed on that date. A signed, time-stamped manifest turns a folder of files into one object whose integrity can be checked with one command. The same idea applied to media is what a C2PA manifest does; a team that can verify one, as described in how to verify content credentials, already understands the other.

Reproducibility: the second-tester test

A record is complete when a second tester, given the record and nothing else, can obtain the same inputs, the same version of the system under test, run the same procedure and get the same result within the stated tolerance. Run this test on your own records before anyone hostile does, using a tester who has only the sealed record. Where the second run differs, the record is missing something: an unpinned dependency, a hand-changed default, a threshold that lived in someone's head.

Some systems cannot reproduce exactly: a detector behind an API may be updated without notice, and a sampling model gives different scores per run. The record must then state the source of variation, report rates with intervals rather than single values, and include enough runs to show the spread. A tolerance fixed in advance, for example that the false positive rate must reproduce within the original interval, belongs in the criteria, alongside the test set discipline in benchmarking AI text detectors.

Worked example: a detector run on 300 documents

A university's assessment office needed a record of a text detector's error rates that would stand up in an appeals hearing. They ran the detector at its production threshold on 300 documents, 150 written under supervision and 150 generated by a language model. The counts are invented but realistic; the record card shows the fields as filed.

Run identifier

DET-2026-09-14-01, sealed manifest hash 3f9c...b21e

Tester and reviewer

Named assessment officer; reviewed by a named second officer on 2026-09-16

System under test

Text detector, vendor build 4.2.1, threshold 0.80, obtained 2026-09-12, API endpoint and account recorded

Inputs

300 documents, fixture v3, each with SHA-256; 150 supervised human essays, 150 generated in logged sessions; list and hashes in inputs.csv

Criteria (fixed 2026-09-10)

False positive rate upper bound below 5 percent; detection rate lower bound above 70 percent; three runs must agree within the interval

Raw outputs

300 scores per run, 3 runs, raw JSON responses stored unedited, hashes in the manifest

Result

False positives 4 of 150, 2.7 percent, 95 percent interval 1.0 to 6.7; detections 119 of 150, 79.3 percent, interval 72.2 to 85.0

Verdict against criteria

Fail on the false positive criterion: the upper bound of 6.7 percent exceeds 5 percent. The detector is not used as sole evidence.

Environment

Named machine, operating system and runtime version; 2026-09-14 09:12 to 10:48 UTC; vendor API

Storage

Write-once bucket with a 7-year retention lock; manifest signed by the office key and time-stamped

The interesting line is the verdict. The point estimate of 2.7 percent met the target most people would set by eye; the criterion, fixed four days before the run, was on the upper bound, and the record shows it failed. A record with the criterion chosen once 2.7 percent was known would have passed, and collapsed under the first question at the hearing.

Common failures that lose a dispute

• Screenshots of dashboards. A screenshot shows a number with no input hashes, no version and no way to reproduce. Export the raw data and hash it.
• Undated or editable exports. A spreadsheet saved last week and editable tomorrow proves nothing about a run last year. Seal exports with a hash and a time-stamp.
• Rerun until green. Repeating a run until it passes and recording only the pass destroys credibility. Record every run, including failures, and fix the rule for combining them in advance.
• Criteria written after the result. A criteria document dated after the run tells a challenger the bar was moved. Time-stamp the criteria before the first input is processed.
• Missing versions. A detector, validator or trust list updated between the run and the dispute makes the record unrepeatable unless versions were recorded.

Retention

Keep the record for as long as the decision it supports can be challenged, plus any period a regulator or policy requires: for academic integrity, often the student's programme plus the appeals window; for editorial and legal disputes, years. Retention covers the whole sealed record, inputs included, since a summary without its inputs cannot be reproduced. Personal data among the inputs stays under the same access controls as other sensitive evidence, with the retention period written into the record at sealing. Planning conventions for the wider suite are in the guide to writing a test plan; the testing AI systems hub holds the guides whose results these records protect.

This guide is part of the testing AI systems hub. It is best read alongside how to verify content credentials and confidence intervals for classifier metrics, which cover the neighbouring questions.

Work it out