Testing a content provenance pipeline: does C2PA survive your CMS?

What a C2PA manifest is, in tester terms

A C2PA manifest is a set of assertions about an asset (the actions taken on it, the ingredients it was made from, a hash of its content) wrapped into a claim, signed by the claim generator, which is the hardware or software that produced it, using an X.509 credential. Manifests live in a manifest store embedded in the file or referenced remotely, and an edited asset can carry a chain of them, each naming its predecessor as an ingredient. The specification calls the arrangement Content Credentials.

Two kinds of binding tie the manifest to the content. A hard binding is a cryptographic hash of the asset's bytes or of a defined region of them; change one pixel and the hash no longer matches. A soft binding is a fingerprint or invisible watermark that can help a validator find a manifest again after the metadata was stripped, and it is weaker by design. The manifest is data your pipeline can discard like any other metadata block. The hard binding means the manifest is valid only for the exact bytes it was computed over, so a resize invalidates it unless the resizer writes a new manifest naming the original as an ingredient.

The test oracle: a valid chain or an honest absence

The oracle is the result a C2PA validator returns for the asset after each stage. The specification defines three manifest states: well-formed (structurally correct), valid (well-formed, unmodified since signing, signature verified and inside its validity period) and trusted (valid, with a signer on the validator's trust list). It also says an asset is valid only when the content bindings still match the bytes. Your oracle accepts three results after a stage: a valid or trusted active manifest whose hash matches the output file; a new manifest signed by the stage itself with the previous manifest as an ingredient; or no manifest at all, reported as such.

Your oracle rejects one result: a manifest that is still embedded but no longer validates, because the bytes changed or the signature fails, while the page or the API reports the asset as carrying credentials. That is the silent broken chain: it looks like provenance, proves nothing, and is the state most likely to mislead a reader or a court. Write this oracle into the plan before the first run.

Inventory every transformation stage

Walk the path with the engineers and list every process that reads and writes the bytes, including the ones outside your codebase. A typical CMS has nine: the upload handler, a virus scanner that may re-save the file, a resizer that produces derivatives, a format converter to WebP or AVIF, an in-browser crop tool, a CDN image optimiser, a social share card generator, an email image proxy and the screenshots people take of the published page. The CDN and the email proxy are where credentials most often vanish without anyone on the team writing a line of code, and which metadata blocks survive which tools is the subject of the guide to metadata survival testing.

Testing a content provenance pipeline: the protocol

1. 1

   Build the fixture

   Collect 40 images with valid manifests from at least two claim generators, in JPEG, PNG and WebP, from 200 KB to 12 MB, some with a chain of two or more manifests. Record the SHA-256 of every file, store the set read-only and confirm every file validates as trusted before you start.

2. 2

   Fix the pass criteria

   Write the criteria from the oracle above, with the number of images and stages, and time-stamp the document. Criteria written after the run are opinions.

3. 3

   Run each stage in isolation

   Feed the 40 originals directly into one stage, capture its output, hash it and run the validator. Classify each result as preserved, re-signed, stripped or broken. Isolation tells you which stage did what.

4. 4

   Run end to end

   Push the same 40 through the whole pipeline as an editor would, including publishing, and validate every rendition: page image, responsive variants, share card, email version.

5. 5

   Check the display layer

   For every rendition, compare what the page shows (a badge, a provenance panel, nothing) with the validator's state for the bytes actually served. A mismatch is a defect whatever stage caused it.

6. 6

   Record the run

   Store the fixture hashes, every output hash, the validator name and version, its trust list version, the pipeline component versions and the time of each step, in the form described in evidence-grade test records.

7. 7

   Automate and gate

   Turn steps 3 to 5 into a job that runs on every change to the image library, the CDN configuration or the list of output formats, with the fixture versioned alongside the code.

Worked example: 40 images through 9 stages

A publisher's team ran the protocol on their CMS with a fixture of 40 signed images, each stage in isolation and then the full chain end to end. The counts are invented but realistic.

Upload and scanning preserved all 40. Resize and conversion stripped all 40, as the library's documentation predicted, and the validator reported no credentials. The crop tool produced the finding that mattered: 12 images passed through untouched because no crop was applied, 22 were stripped, and 6 PNG files kept their manifest store while the pixels changed. Those 6 validated as invalid, and the article template showed a credentials badge on all 6. The CDN optimiser preserved 31 and stripped the 9 files above its 400 KB re-encoding threshold. Share card, email proxy and screenshot produced no credentials, as the criteria allowed. End to end, 0 of 40 reached the page with a valid manifest, because resize sits ahead of everything else.

Against the criteria written before the run, the pipeline failed twice: the crop tool created broken manifests, and the display layer showed a badge for assets whose bytes did not validate. The stripping at resize went to the product owner as a decision, since no stage re-signed derivatives, and a manifest that never reaches the reader protects nobody.

Regression, reporting and what to publish

A provenance pipeline breaks silently. A library upgrade, a CDN setting or a new output format can remove or corrupt manifests with every functional test green, because nothing in a functional suite looks at the manifest store. The fixture run therefore belongs in the pipeline itself, alongside the checks described in continuous testing: version the fixture with its hashes, pin the validator version and trust list, and fail the build when any stage moves from preserved or re-signed to broken, or when the display layer disagrees with the validator.

Report per stage: counts out of the fixture size, the four-way classification, component versions and validator version. State which stages drop credentials by design so that editors know what readers will see. How a reader or a fact checker should then interpret the served file is covered in how to verify content credentials, and the rest of the testing AI systems hub covers the detection side of the same problem.

This guide is part of the testing AI systems hub. It is best read alongside how to verify content credentials and metadata survival testing, which cover the neighbouring questions.