Evidence-grade test records for authenticity systems

When a test record becomes evidence

Most test records are read once, by the team, deciding whether to ship. A record about an authenticity system has a second audience that arrives later and reads it adversarially: a student appeals panel quoting a false positive rate, a photographer disputing a correction, a court receiving a verification session as an exhibit. That reader wants a reason to disbelieve the result, and an ordinary test report offers several: no hashes, no versions, criteria apparently written after the numbers were known.

The test documentation standard, ISO/IEC/IEEE 29119-3, already asks for most of the right content: completion criteria, test data, environment, execution logs and incident reports. Evidence grade adds integrity and reproducibility. Those techniques come from digital forensics, where NIST SP 800-86 and RFC 3227 describe collecting and preserving evidence for incident response, and they apply almost unchanged to test artefacts.

What makes evidence-grade test records different

The eight field groups of an evidence-grade test record and the challenge each one answers
Field group	What to record	Why a challenger asks for it
Who	Tester's name and role, reviewer's name, organisation	Independence and competence
System under test	Product, exact version or build hash, configuration, threshold, model identifier, date obtained	Whether the tested system is the one that produced the disputed result
Inputs	Every input file with its SHA-256, its source, how labels were established, fixture version	Whether the inputs were what the record says, and whether they were cherry-picked
Criteria	The pass criteria, metrics and confidence level, with the time-stamp of the document that fixed them	Whether the bar was set before or after the result
Outputs	Raw outputs per input (scores, validator status codes, logs), unedited, with their hashes	Whether the summary matches the data
Time	Start and end of every run in UTC, and the time of each hand-off	Sequence, and whether the system changed between runs
Environment	Machine, operating system, runtime, network conditions, validator or tool versions and trust list versions	Whether the result depends on something other than the system under test
Integrity	A manifest listing every artefact with its hash, itself hashed, signed and time-stamped, and the storage location	Whether anything was altered after the run

None of these fields is exotic. In an evidence-grade record every field is present for every run, inputs are identified by hash, and the record is sealed rather than saved. Teams that already report detector results with intervals, as in confidence intervals for classifier metrics, usually have the numbers and lack the integrity layer.

Chain of custody for test artefacts

Chain of custody is a log of who held an artefact, when, with proof that it did not change while they held it; for digital artefacts the proof is a hash. Hash every input before the run and every output immediately after, and record them in the run log. Store the artefacts in write-once storage, where objects cannot be overwritten or deleted for a retention period. Whenever an artefact is copied or handed to someone else, log the hash before and after the hand-off.

Then seal the record: write a manifest listing every artefact with its hash, hash that manifest, sign it with a key held by a named person or the organisation, and obtain a time-stamp from a time-stamp authority so that the signature can be shown to have existed on that date. A signed, time-stamped manifest turns a folder of files into one object whose integrity can be checked with one command. The same idea applied to media is what a C2PA manifest does; a team that can verify one, as described in how to verify content credentials, already understands the other.

The custody question

For any number in the report, can you name the file it came from, the hash of that file, who has held it since the run and the proof that it is unchanged? If not, the number is testimony, not evidence.

Reproducibility: the second-tester test

A record is complete when a second tester, given the record and nothing else, can obtain the same inputs, the same version of the system under test, run the same procedure and get the same result within the stated tolerance. Run this test on your own records before anyone hostile does, using a tester who has only the sealed record. Where the second run differs, the record is missing something: an unpinned dependency, a hand-changed default, a threshold that lived in someone's head.

Some systems cannot reproduce exactly: a detector behind an API may be updated without notice, and a sampling model gives different scores per run. The record must then state the source of variation, report rates with intervals rather than single values, and include enough runs to show the spread. A tolerance fixed in advance, for example that the false positive rate must reproduce within the original interval, belongs in the criteria, alongside the test set discipline in benchmarking AI text detectors.

Worked example: a detector run on 300 documents

A university's assessment office needed a record of a text detector's error rates that would stand up in an appeals hearing. They ran the detector at its production threshold on 300 documents, 150 written under supervision and 150 generated by a language model. The counts are invented but realistic; the record card shows the fields as filed.

Figure 2Detector error rates on 300 documents with 95 percent Wilson bounds (worked example)

Invented but realistic counts: 4 false positives in 150 human documents and 119 detections in 150 generated ones. The record reports the bounds, because a challenger will ask what the 2.7 percent could really be.

Run identifier: DET-2026-09-14-01, sealed manifest hash 3f9c...b21e
Tester and reviewer: Named assessment officer; reviewed by a named second officer on 2026-09-16
System under test: Text detector, vendor build 4.2.1, threshold 0.80, obtained 2026-09-12, API endpoint and account recorded
Inputs: 300 documents, fixture v3, each with SHA-256; 150 supervised human essays, 150 generated in logged sessions; list and hashes in inputs.csv
Criteria (fixed 2026-09-10): False positive rate upper bound below 5 percent; detection rate lower bound above 70 percent; three runs must agree within the interval
Raw outputs: 300 scores per run, 3 runs, raw JSON responses stored unedited, hashes in the manifest
Result: False positives 4 of 150, 2.7 percent, 95 percent interval 1.0 to 6.7; detections 119 of 150, 79.3 percent, interval 72.2 to 85.0
Verdict against criteria: Fail on the false positive criterion: the upper bound of 6.7 percent exceeds 5 percent. The detector is not used as sole evidence.
Environment: Named machine, operating system and runtime version; 2026-09-14 09:12 to 10:48 UTC; vendor API
Storage: Write-once bucket with a 7-year retention lock; manifest signed by the office key and time-stamped

The interesting line is the verdict. The point estimate of 2.7 percent met the target most people would set by eye; the criterion, fixed four days before the run, was on the upper bound, and the record shows it failed. A record with the criterion chosen once 2.7 percent was known would have passed, and collapsed under the first question at the hearing.

Common failures that lose a dispute

Screenshots of dashboards. A screenshot shows a number with no input hashes, no version and no way to reproduce. Export the raw data and hash it.
Undated or editable exports. A spreadsheet saved last week and editable tomorrow proves nothing about a run last year. Seal exports with a hash and a time-stamp.
Rerun until green. Repeating a run until it passes and recording only the pass destroys credibility. Record every run, including failures, and fix the rule for combining them in advance.
Criteria written after the result. A criteria document dated after the run tells a challenger the bar was moved. Time-stamp the criteria before the first input is processed.
Missing versions. A detector, validator or trust list updated between the run and the dispute makes the record unrepeatable unless versions were recorded.

Retention

Keep the record for as long as the decision it supports can be challenged, plus any period a regulator or policy requires: for academic integrity, often the student's programme plus the appeals window; for editorial and legal disputes, years. Retention covers the whole sealed record, inputs included, since a summary without its inputs cannot be reproduced. Personal data among the inputs stays under the same access controls as other sensitive evidence, with the retention period written into the record at sealing. Planning conventions for the wider suite are in the guide to writing a test plan; the testing AI systems hub holds the guides whose results these records protect.

Common questions

What is an evidence-grade test record?

A test record complete enough that a hostile reader can verify every claim from it alone: who ran it, hashed inputs, exact versions, criteria fixed in advance, raw outputs, timestamps and environment, sealed against later change.

How do I keep chain of custody for test artefacts?

Hash every input and output, store artefacts in write-once storage, log every hand-off with hashes before and after, and seal the whole record with a signed, time-stamped manifest listing every file and its hash.

Why must pass criteria be fixed before the run?

Because a criterion chosen after the numbers are known can always be chosen to pass. A criteria document time-stamped before the run is the only defence.

What if the system under test cannot be reproduced exactly?

State the source of variation, run several times, report rates with confidence intervals, and fix a reproduction tolerance in the criteria. Record the exact version and date, since a hosted system may change.

How long should test records be retained?

For as long as the decision they support can be challenged, plus any regulatory or policy period, with the retention rule and personal data controls written into the record at sealing.