Reproducibility: test results with provenance you can rerun

Three words that get confused

The ACM artifact review policy defines three terms worth keeping apart. Repeatability: the same team obtains the same measurement on the same setup, with stated precision, on multiple trials. Reproducibility: a different team obtains it with the same procedure and system, using the original artifacts. Replicability: a different team obtains it with artifacts built independently. The policy is explicit that exact agreement is not expected; results must agree within a tolerance suited to the experiment, and the differences must not change the main claim.

For a tester the three form a ladder: repeat it yourself, then let a colleague reproduce it from your files, then see it replicated on a fresh set. Most reports never say which rung they reached.

Why AI evaluations resist reruns

Classical software is deterministic enough that a rerun failure points at the code or the environment. Language models and detectors break that assumption five ways. The model changes silently when a hosted model behind a fixed name is retrained or replaced. Sampling above temperature zero makes each generation a draw from a distribution, so two runs differ by design. Inference is non-deterministic even at temperature zero on some serving stacks, because batching and floating-point reduction order move the logits at the margin. The evaluation set moves when someone adds a case without bumping the version. And the threshold drifts when a vendor recalibrates the score-to-verdict mapping.

A false positive rate falling from 8 of 200 to 3 of 200 could then be a better model, a smaller set, a moved threshold or three lucky draws. Nobody can say which without pinned inputs; the sampling side is covered by intervals around the rate you measured.

What to pin, and what happens when you do not

Structuring a run for reproducibility

1. 1

   Freeze the inputs before the first item runs

   Write every field in the table into a manifest and time-stamp it. A manifest written afterwards records what you think happened.

2. 2

   Run from the manifest, not from memory

   The harness reads every parameter from the manifest and refuses to start if a field is missing.

3. 3

   Sample deliberately

   Where temperature must be above zero, fix the seed, run each item five or ten times, and report the per-item majority or mean with its spread.

4. 4

   Hash every raw output as it lands

   Store per-item outputs and verdicts, hash each file and append the hashes to the manifest. Rates are computed afterwards from those files.

5. 5

   Compute rates with intervals

   Report every rate with its denominator and a confidence interval; the interval is the tolerance a rerun must land inside.

6. 6

   Hand the manifest to a second team

   Someone who did not run the first pass reruns from the manifest alone, on a different machine, and files their rate and interval against yours.

Worked example: 24 reruns by what was pinned

A team benchmarking a text detector took one published result, a false positive rate of 11 of 240 human-written documents (4.6%, 95% interval 2.3% to 8.1%), and had a second team rerun it 24 times under four levels of pinning, six reruns per level. The counts are invented but realistic.

Leaving the model to float cost two reruns: the hosted model had been updated in between and the newer version flagged fewer human documents. Leaving the set to float cost four: a colleague had removed 19 short documents as too easy, and the denominator dropped to 221 with most of the false positives still in it. With nothing pinned, the one agreeing rerun agreed by luck. The team's rule afterwards: a result without a manifest is filed as a note. Keeping a set fixed is covered in designing an evaluation set that stays fixed.

When the rerun disagrees

A rerun outside the interval is a finding, and the response is a diff rather than a debate. Compare the two manifests field by field. If they differ, the difference is the cause until shown otherwise. If they are identical and the rates still differ, the system is non-deterministic beyond what the seed controls, itself worth reporting. Rerun a third time on the original machine to separate machine from model effects, report all three rates with intervals, and never average the reruns together.

Flaky tests are the same failure under another name: a test that passes once and fails next time with no code change has an input nobody pinned, whether time, thread order, a network call or sampling. The disciplines in the guide to tracking down tests that fail intermittently apply directly: quarantine, isolate the dependency, then pin or fake it. Evaluation work adds one more loose input, the model version, and it is the one most often left floating.

Pitfalls

• Pinning the name and not the version. A model name is a pointer, and what it points to changes. Record a version string, snapshot date or weights hash; if the vendor offers none, record the date and say so.
• Hashing the set once and never again. The set's hash belongs in every manifest, recomputed at run time. A January hash proves nothing about June's files.
• Treating temperature zero as deterministic. Some serving stacks still vary at the margin; run the same item three times before claiming it.
• Aggregating before storing. A rate stored without its per-item outputs cannot be recomputed, and a disagreeing rerun cannot be traced to the items.
• Reporting the rate without the interval. A rerun then has no pass criterion. 4.6% of 240 and 4.6% of 24 are different claims; only the interval says so.

What to record

The manifest is the deliverable: the pinned inputs from the table, the hashes of every raw output, the rate with its denominator and interval, the machine, the time in UTC and the person who ran it. Store it read-only beside the outputs with the integrity treatment described in keeping test records that hold up under challenge: a hash of the manifest and, where the result may be contested, a signature or trusted time-stamp over that hash. A second team's manifest is filed beside the first with a one-line verdict. The format in building a detector benchmark that others can check assumes this manifest exists, and the other guides on testing AI systems lean on it whenever a rate is quoted.

Reproducibility is the difference between a number and a measurement. Pin the model, prompt, seed, set, validator and threshold; hash what went in and what came out; report the rate with its interval; and let someone who was not there rerun it from the manifest alone.

This guide is part of the testing AI systems hub. It is best read alongside evidence-grade test records for authenticity systems and confidence intervals for classifier metrics, which cover the neighbouring questions.