Where the idea comes from
Chen, Cheung and Yiu described metamorphic testing in 1998 as a way to test programs whose correct output is hard to compute, such as numerical simulations. If you cannot say what sin(x) should be to twenty places, you can still say that sin(x) must equal sin(pi minus x). That shift from checking values to checking relations is exactly what the oracle problem in AI testing calls for. Ribeiro and colleagues applied the same thinking to language models in their CheckList work, organising tests as invariance, directional and minimum-functionality checks.
Three families of relation
| Family | Relation | Example for a text classifier | Example for an AI detector |
|---|---|---|---|
| Invariance | Output must not change | Swapping one person's name for another leaves the sentiment label unchanged | Changing a city name in a human essay does not change the verdict |
| Direction | Output must move one way | Adding a strong negation lowers the positive score | Appending a clearly machine-written paragraph does not lower the machine score |
| Consistency | Outputs must be compatible | A document and its paragraphs get compatible labels | A document's score is not below the scores of all its halves |
Good relations come from the specification and from domain knowledge, which is why testers are well placed to write them. The discipline is the same as equivalence partitioning: decide which changes to the input should be irrelevant to the output, and which should matter, then test both sides.
Running metamorphic tests at scale
- Write each transformation as a small function that turns a source case into a follow-up case, and check a sample of pairs by eye to make sure it does what you think.
- Apply each relation to at least 200 source cases drawn from the real input population, so the violation rate means something.
- Fix the model settings for both runs; with sampling enabled, run each input several times and compare distributions rather than single outputs.
- Report the violation rate per relation with a confidence interval, and keep failing pairs as regression cases.
A relation that is violated 2 percent of the time is not a flaky test. It is a measured property of the model, and the number should go into the release report.
Worked example: four relations on a ticket classifier
A support desk uses a language model to assign a priority (low, normal, high) and a product area to incoming tickets. The team wrote four relations and ran them on real tickets with personal data removed. The counts are invented but realistic; the intervals are 95 percent Wilson intervals.
| Relation | Transformation | Pairs | Violations | Rate and interval |
|---|---|---|---|---|
| Invariance | Swap the customer's name for another name | 400 | 9 | 2.3% (1.2 to 4.2) |
| Invariance | Insert two typos into words that are not product names | 400 | 31 | 7.8% (5.5 to 10.8) |
| Direction | Append a sentence reporting a full outage; priority must not fall | 250 | 14 | 5.6% (3.4 to 9.2) |
| Consistency | Classify the ticket and its first paragraph alone; product area must agree | 300 | 27 | 9.0% (6.3 to 12.8) |
The typo result changed the release. Most violations turned out to be tickets where a typo landed in a word the model had been using as a product cue, which pointed to a prompt that leaned on exact keywords. The direction relation found 14 tickets where reporting an outage lowered the priority, which no labelled test would have caught because nobody had thought to label that case. The team fixed the prompt, reran the four relations on the same 1,350 pairs, and recorded the new rates beside the old ones: 2.0, 2.5, 1.6 and 8.7 percent. The consistency rate barely moved, and reading its violations showed why: in most of them the first paragraph genuinely did not say which product was affected. That relation was rewritten to apply only to tickets whose first paragraph names a product, which is a change to the specification of the test rather than to the model.
The cost of the run is worth stating. 1,350 pairs is 2,700 inputs, each run three times because sampling was on, so 8,100 model calls per candidate. At about one second per call that is a little over two hours on a single connection, which is cheap enough to run on every prompt change and far cheaper than labelling 1,350 tickets by hand.
Common mistakes when writing relations
- Writing a relation that is not actually true. Adding a negation does not always lower sentiment ("not bad at all"); check a sample of pairs by hand before trusting the violation count.
- Producing follow-up inputs that are broken. A transformation that leaves ungrammatical text tests robustness to noise, which is a different relation from the one you meant.
- Setting a tolerance so tight that sampling noise counts as violation. With sampling on, run each input several times and compare distributions, or fix the seed and temperature for the run.
- Counting each violating pair as a defect. The number that matters is the rate per relation on a fixed set; single pairs go into the regression list.
- Reusing the relation pairs to tune the prompt, so that the next run passes because the model has seen the cases.
- Stopping at invariance. Direction and consistency relations are where models that pass surface checks still fail, and they are the ones that need domain knowledge to write.
How to report a metamorphic run
Report each relation on its own line: the relation and its transformation, the version of the transformation code, the source set identifier, the number of pairs, the number of violations, the rate with its interval, and the model, prompt and sampling settings for both runs. Attach a sample of violating pairs so that a reader can see what the violations look like rather than trusting a percentage. Rates from two runs are only comparable when the source set and the transformation version are the same, so state both. If a violation rate is ever used to support a claim about a product's behaviour, keep the whole run as an evidence-grade test record, with the pairs and the outputs preserved.
What metamorphic testing cannot tell you
A model can satisfy every relation and still be wrong in the same way on both inputs. Metamorphic tests find inconsistency, not error. Pair them with a labelled evaluation set for accuracy and with human review for quality. They are strongest as a regression net: once a relation is written, it runs on every model, prompt or data change at almost no cost, which makes it a natural part of prompt regression testing.
Common questions
What is metamorphic testing?
A technique that checks how the outputs of two related inputs must relate, instead of comparing one output with a known expected value. It was introduced for programs whose correct output is hard to compute.
Why is it useful for language models?
Because language features rarely have a single correct output, but testers can often say what must stay the same or move in one direction when the input changes.
What are examples of metamorphic relations for text?
Renaming a person should not change a sentiment label; adding a negation should lower a positive score; a question asked in two ways should get compatible answers.
How many test pairs are needed?
At least a few hundred per relation, drawn from real inputs, so that the violation rate can be reported with a meaningful confidence interval.
Does passing metamorphic tests mean the model is correct?
No. It means the model is consistent under those transformations. It can still be wrong on both inputs, so combine it with labelled evaluation.
How is metamorphic testing different from property-based testing?
Property-based testing checks that one output satisfies a property, such as a valid schema. Metamorphic testing checks a relation between two outputs from related inputs. The two combine well: a property check on each output, then a relation check across the pair.
Sources
This guide is part of the testing AI systems hub. It is best read alongside prompt regression testing and how to test ai systems, which cover the neighbouring questions.