How to test AI systems: oracles, evaluation sets and risk

Why the oracle problem is the whole problem

Testing research calls the difficulty of deciding whether an output is correct the oracle problem, and Barr, Harman, McMinn, Shahbaz and Yoo's survey treats it as one of the oldest open questions in the field. For most business software the oracle is cheap: the specification says a discount of 10 percent on 200 is 20. For an AI component the oracle is expensive or absent. A summariser can produce many good summaries of the same text; a classifier is expected to be wrong some of the time; a ranking has no single right order. That does not make the component untestable. It moves the tester's work from writing expected values to designing measurements.

The practical consequence is that a single test case proves very little. One correct answer from a text classifier tells you nothing about its error rate, in the same way that one fast response tells you nothing about the p95 latency in a performance test. The unit of evidence becomes a sample: a few hundred cases per slice of the input space, scored against a criterion that is written down before the run.

Three layers of testing

The first layer is where most production incidents actually come from, and it needs nothing new. A model that returns malformed output, a timeout that is not handled, a retry loop that doubles cost: these are ordinary defects found with ordinary black-box and white-box techniques. Treat the model as an unreliable external dependency and test the code that calls it the way you would test code calling any flaky service.

Model behaviour: rates, not verdicts

The second layer measures how often the model is right, and on which inputs. The artefact is an evaluation set: labelled cases the model has never seen, stratified by the slices that matter to the risk (language, length, domain, user group). The result is a confusion matrix and its derived rates, each reported with a confidence interval so that a reader can tell a real difference from noise. Designing that set well is a craft of its own, covered in the guide to evaluation test set design.

Write the pass criterion as a rate with a bound, before the run: for example, the false positive rate on human-written text must be below 1 percent, with the upper end of the 95 percent confidence interval below 2 percent, on every language slice with at least 300 cases. A criterion written after the results are in tends to be written around them.

Properties: metamorphic and invariance tests

Metamorphic testing, introduced by Chen, Cheung and Yiu in 1998 and surveyed by Segura and colleagues in 2016, sidesteps the oracle by checking relations between outputs instead of the outputs themselves. You may not know the correct sentiment score for a review, but you know it should not change when a person's name in the review changes, and it should fall when a positive word is replaced by a negative one. Ribeiro and colleagues' CheckList method organises these into three kinds of test: minimum functionality tests (simple cases the model must get right), invariance tests (changes that must not move the output) and directional expectation tests (changes that must move it one way).

1. 1

   Name the property

   State what must hold whatever the exact output: a classifier's verdict does not depend on the author's name; a summary never introduces a figure absent from the source; a detector's verdict does not flip when line breaks change.

2. 2

   Write the transformation

   Define the input change that exercises the property: swap names from a list, reformat whitespace, translate and back-translate, truncate to 80 percent.

3. 3

   Generate pairs at volume

   Apply the transformation to a few hundred seed cases so that the failure rate, not a single example, is the result.

4. 4

   Set the tolerance

   Decide how often a violation is acceptable. Zero for safety properties; a small stated rate for soft ones.

5. 5

   Keep them as regression tests

   Every property test goes into the suite that runs on each model, prompt or data change.

Risk tiers decide the depth

Not every AI feature needs the same rigour. Risk-based testing applies unchanged: score likelihood and impact per feature and let the score set the depth. The NIST AI Risk Management Framework and the EU AI Act both organise obligations by the harm a system can cause, and a tester can borrow the same logic. A feature that suggests a subject line needs a sample check and property tests. A feature that flags a person, a document or a transaction as suspicious needs a full evaluation per slice, a documented false positive ceiling, a human review path and monitoring, because its errors land on people.

Monitoring is part of the test plan

A model evaluated in September meets a different population in March. Inputs drift, users adapt, and a component that sits behind a prompt changes whenever the prompt does. Plan for it in the test plan: sample live traffic for periodic human labelling, track the flag rate and the disagreement rate between the model and reviewers, and re-run the full evaluation on any change to the model, the prompt or the preprocessing. The guide to testing LLM features covers the regression suite that makes those re-runs cheap.

This guide is part of the testing AI systems hub. It is best read alongside how to benchmark an ai text detector and evaluation test set design for ai models, which cover the neighbouring questions.