Chain of custody for test evidence: logs to signed reports

What chain of custody means for a tester

Chain of custody is a term from forensic practice: a documented, unbroken record of possession and handling for an item of evidence, from creation to presentation. NIST SP 800-86 frames the record as the answer to four questions at every step: who handled the item, when, what was done to it, and how its integrity was preserved. Test evidence needs the same treatment when a result can be contested, whether the item is a screenshot, a log, a packet capture, a dataset, a detector report or a C2PA validator's output. The record has two layers: a hash that proves the bytes are unchanged, and a log that proves who held them.

Why a test result needs one

Two disputes recur. A vendor says a benchmark test set was altered after the run; the record shows the same dataset hash before the run and in the report. A platform team says the fixture behind a broken-manifest finding was corrupted in transit; the record shows its hash at creation, upload and validation. The surrounding document is the subject of the guide to records a hostile reader can check; this guide covers the chain those records depend on.

The six elements

Hash at creation. Compute a SHA-256 digest as soon as the producing tool writes the item, before any copy, upload or rename. A hash computed an hour later proves only that nothing changed after that hour. Custody log. Append one entry per event: item identifier, hash, handler, action, and a timestamp from a clock synchronised to a common source. NIST SP 800-92 treats synchronisation as a precondition for log evidence: logs with unrelated clocks cannot be ordered. Read-only storage. Keep the original where overwrites and deletions are rejected, such as a locked object store, and re-check the hash on every read.

Named handlers. Every event names a person or a specific automated identity; an entry saying the QA account exported the file does not say who did. Documented transformations. A cropped screenshot, a filtered log or a redacted export is a new item with its own hash, its own log entry and a link to the parent's hash, and the parent is kept. Signed or time-stamped report. The report lists every hash it relies on and is signed with a named key or time-stamped with an RFC 3161 token, which binds its hash to a time asserted by a third party.

How to set up a chain of custody for test evidence

1. 1

   Decide what counts as evidence

   Before the run, list the items whose integrity might be questioned: fixtures, raw outputs, screenshots, captures, exports. Put the list in the plan, following the template for a test plan that survives review.

2. 2

   Synchronise the clocks

   Point every producing machine at one time source, record each offset at the start of the run, and log in UTC. A five-minute drift between hosts turns a clear sequence into a dispute.

3. 3

   Hash at creation, automatically

   Wrap the capture and export tools so the digest is computed in the step that writes the file. A post-run script proves only the folder's state when it ran.

4. 4

   Write the custody log as an append-only stream

   Each entry: item identifier, SHA-256, handler identity, action, UTC time, parent hash if any. Store it where entries cannot be edited, and hash the log after each session.

5. 5

   Store originals read-only and derive from copies

   Upload originals to a write-once location whose reader verifies the hash before returning bytes. When a copy is cropped or filtered, record the tool, parameters, new hash and parent hash.

6. 6

   Sign or time-stamp the report

   List every cited hash in the report, then obtain an RFC 3161 token for the report's own hash or sign it with a named key, and store the token with the report.

Worked example: one release, 312 evidence items

A team testing a document ingestion service with an AI text detector and a signed image pipeline counted its evidence items from one release and how many kept an unbroken chain. The counts are invented but realistic.

Of 312 items, 281 kept an unbroken chain (90.1%, 95% interval 86.2% to 93.0%). Twenty-two of 140 screenshots had been cropped before attachment to defect reports and the originals overwritten, so the creation hash matched no surviving file. Five of 14 packet captures came from a laptop whose clock was 11 minutes behind the log server; they were intact but could not be placed in sequence. Four detector reports had been reformatted and the raw output deleted.

The fix for screenshots was a capture tool that writes the original to the read-only store and hands the tester a working copy; for captures, a clock check before every run. The 31 broken items stayed in the report, labelled unsupported, since removing them would itself be an undocumented change.

Pitfalls that break the chain

• Hashing after the fact. A digest computed at archiving proves the state at archiving. Everything before it is unrecorded, and that is the window a challenger asks about.
• Overwriting the original. Cropping, redacting or filtering in place destroys the item the creation hash refers to. Derived items need their own hash and a surviving parent.
• Shared accounts and unnamed automation. An entry naming a team account or a generic bot does not answer who. Give each job its own identity and each person their own login.
• Trusting the summary. A dashboard, a spreadsheet or a formatted PDF is a transformation of the raw output. If the raw file was discarded, the summary rests on nothing.

What to record and how to report it

The report cites items rather than describing them from memory: for each, the identifier, the SHA-256, the creator, the UTC creation time and the storage location, plus the parent hash and transformation for derived items. Attach the custody log, or its hash, and the report's RFC 3161 token or signature. For a provenance check, include the fields listed in a verification session that names its validator. ISO/IEC/IEEE 29119-3 gives this material a place in the test completion report, and the share of items with an intact chain, with denominator and interval, belongs with the metrics a test manager tracks each release.

State the gaps rather than curating them away. The hub for authenticity and detector testing covers the measurements these records protect, and the guide to pushing signed images through every CMS stage shows a fixture run where the custody chain is the whole test.

A chain of custody turns a folder of screenshots, logs and exports into evidence that a supplier, an auditor or a court can check without trusting the person who ran the test. Hash at creation, log every handler, store originals read-only, and time-stamp the report.

This guide is part of the testing AI systems hub. It is best read alongside evidence-grade test records for authenticity systems and testing a content provenance pipeline, which cover the neighbouring questions.