Prompt regression testing: baselines, diffs and release gates

Why prompts need regression tests

A prompt is code written in a natural language, executed by a component whose behaviour changes when its supplier updates it. The two ways it breaks are familiar from any software: someone changes it to fix one thing and breaks another, or a dependency changes underneath it. What is new is that outputs vary from run to run, so an exact-match assertion fails constantly and teams give up on testing. The fix is to test the properties that matter and measure them as rates, the approach laid out in testing language model features.

Build the regression suite in six steps

1. 1

   Collect the suite from real use

   Sample real inputs (with personal data removed), add every past failure, and add edge cases chosen by partitioning: very long inputs, empty inputs, other languages, and hostile instructions of the kinds catalogued in the OWASP list of language model application risks.

2. 2

   Write layered checks per case

   Exact checks for structure (valid JSON, required fields, length limits), property checks for content (the answer cites the provided document, no invented product names), and a graded quality score only where the first two cannot reach.

3. 3

   Validate the grader

   If a model grades quality, compare its grades with human grades on 200 or more cases and measure agreement before trusting it, as in inter-rater agreement for evaluation labels. Zheng and colleagues found that model graders can match human preferences well but show position and verbosity biases.

4. 4

   Record a baseline

   Run the suite on the current prompt and model three or more times per case if sampling is enabled, and store outputs, check results and settings under a version tag.

5. 5

   Run on every change and compare rates

   Any change to prompt text, model version, retrieval data or sampling settings triggers the suite. Compare pass rates per check family against the baseline with confidence intervals.

6. 6

   Gate the release

   Block the change if a gated rate falls by more than the agreed margin, and review every newly failing case by hand before deciding it is noise.

Handling variation without hiding regressions

When outputs are sampled, a single run per case is a coin toss. Run each case several times and treat the case as passing if it passes in an agreed share of runs, or compare the distribution of scores. Track cases that flip between runs separately, as you would flaky tests, because instability is itself a finding. Relations from metamorphic testing are especially useful here, since they check consistency directly.

Worked example: the gate decision on candidate 3

Candidate 3 in the figure above was a prompt change meant to make answers shorter. The suite has 600 cases, each run three times; a case passes a check if it passes in at least two of three runs. The gate allows a 2 point drop in any pass rate. The counts are invented but realistic.

The 45 newly failing property cases were reviewed by hand. 38 of them had dropped the quoted passage to meet the new length instruction, which is a real regression caused directly by the change; 7 were borderline cases that flip between runs and were moved to the unstable list. The candidate went back with a revised instruction that protected the citation, and candidate 4 passed. Without the property layer the release would have shipped on a graded score that moved by a tenth of a point.

Common mistakes in prompt regression testing

• Running each case once with sampling on, then arguing about whether a failure is real. Fix the runs-per-case rule and the pass rule in the plan.
• Letting the grader be the only check. Graders drift with their own model updates and carry length and position biases; put structure and property checks first.
• Gating on an overall pass rate. A 7 point drop in one family disappears inside a 600-case average.
• Rerunning the baseline on the new model. The baseline is a recorded artefact from the previous version, and regenerating it hides the change you are testing for.
• Adding failing cases to the suite and then tuning the prompt on them until they pass, which turns a regression suite into a training set.
• Skipping the run for supplier-side model updates because nobody on the team changed anything.

Expert tips

• Tag every case with the failure it was added for. When a case fails again, the tag tells the reviewer what the original bug was without a search.
• Keep a small, fast smoke subset of 50 cases for local iteration and run the full suite in the pipeline; teams that only have the full suite stop running it.
• Record the supplier's model identifier from the response, not from configuration, so that a silent server-side change shows up in the run record.
• Review a random sample of passing cases after each run as well as the failures. A prompt can pass every check with answers that have quietly become worse.
• Store the unstable list as a first-class output and watch its size. Growth in unstable cases usually precedes a regression that the gate will catch later.

How to report a regression run

Each run should be reported as a comparison, never as a single score: the baseline version and the candidate version, the suite version, the runs per case and the pass rule, the per-family pass rates with counts and intervals, the gate margin, the list of newly failing cases with a one-line diagnosis for each, and the decision. Keep the outputs of both runs, since a newly failing case is only diagnosable by reading the two answers side by side. The baseline is the most valuable record the team owns and should be stored with the same care as a release build. The guide to testing language model features covers the wider suite this run sits in.

This guide is part of the testing AI systems hub. It is best read alongside metamorphic testing for language models and test design for llm features, which cover the neighbouring questions.