Evaluation test set design for AI models

Start from the decision, not the data you have

A test set exists to support a decision: ship this model, switch to that one, set this threshold, publish this benchmark. Write the decision and the metric that informs it before collecting a single case. If the decision is whether a text classifier's false positive rate stays below 1 percent for every writer group, the set needs enough human-written cases in each group to bound that rate; machine-written cases matter less for that particular question. This is the same discipline as the test strategy: scope and risk first, cases second.

Partition the input space into slices

Equivalence partitioning transfers directly. Divide the inputs into classes the model may treat differently and report each class separately: language and language background, length band, genre or domain, source channel, edited versus raw, and any group whose errors carry legal or ethical weight. Boundaries matter as well: very short inputs, inputs at the model's length limit, mixed-language text, formatting the training data rarely contained.

Slice dimensions for a text model evaluation; pick the ones your risk analysis names
Slice dimension	Example partitions	Why it is reported separately
Author background	native, other first language, early learner	Error rates can differ sharply between groups
Length	under 150 words, 150 to 600, over 600	Short inputs give noisy scores
Genre	narrative, technical, formulaic templates	Predictable genres look machine-like
Processing	raw, edited by a person, paraphrased	Real inputs are rarely raw
Time	before and after the model's training cut-off	Recent topics expose staleness and leakage
Adversarial	deliberate evasion, prompt injection, odd encodings	Measures robustness, not typical accuracy

Size each slice from the interval you need

The number of cases follows from how precisely a rate must be known. For a proportion near p, the half-width of a 95 percent interval is about 1.96 times the square root of p(1 minus p)/n, so the cases needed grow with the inverse square of the precision. To know a false positive rate near 1 percent to within half a point takes about 1,500 cases in that slice; to within a quarter point, about 6,000. When a slice shows no errors, the rule of three bounds the rate below 3/n, so 300 clean cases bound it below 1 percent. Report the rates with intervals so that a reader can see which slices are adequately sized.

Leakage and contamination

Kapoor and Narayanan's review of machine-learning-based science found data leakage behind errors in hundreds of published papers across many fields, and generative models add a new form: benchmark items that appeared in the pretraining data, which Sainz and colleagues argue must be measured for every benchmark. For a test team the checks are concrete.

Split by source, not by row. If documents from the same author, course or thread appear on both sides of a train and test split, the model is tested on its memory of that source.
Remove duplicates and near-duplicates across every set the model or its prompts have touched, using normalised text and fuzzy matching, not only exact hashes.
Respect time. Keep a slice of cases created after the model's training cut-off; a large drop on that slice suggests the older cases were partly memorised.
Hide a slice. Keep one slice that nobody on the team looks at until the release decision, and replace it each quarter, so that prompt and threshold tuning cannot overfit it.

Test lead's rule

Every time someone looks at a test case to improve a score, that case moves one step closer to being training data. Budget for a hidden slice that is looked at once.

Labels are measurements too

A label is an annotator's judgement, and annotators disagree. Have at least two people label each case independently on a sample, report their agreement with Cohen's kappa, and resolve disagreements with a written rule. For provenance questions such as whether a text was written by a person, the label should come from how the case was produced (supervised writing, generation logs), not from someone's impression of the text, because impressions are exactly what the model under test is trying to replicate.

Version and document the set

Treat the test set as a test asset under configuration control. Give it a version number, store it with the scripts that score it, and never change cases inside a version. Write a datasheet in the sense of Gebru and colleagues: why the set exists, how each slice was collected, who labelled it, known gaps, and what it must not be used for. The detector benchmark protocol and the LLM feature regression suite both depend on a set built this way.

Common questions

How big should an AI evaluation set be?

Size each slice from the precision you need, not the whole set from a round number. A few hundred cases per slice bounds a rare error rate; distinguishing two models whose rates differ by one point needs thousands per slice.

What is data leakage in model evaluation?

Any path by which information from the test cases reaches the model or the people tuning it: shared sources across splits, duplicates, cases used to design prompts, or benchmark items present in pretraining data. Leakage inflates results.

What is benchmark contamination?

The presence of benchmark items, or close copies, in a model's training data. Measure it by testing on cases created after the training cut-off and by searching for near-duplicates where the training data can be inspected.

Should the test set match the real input distribution?

Keep one slice that mirrors production for an overall estimate, and report the other slices separately. A purely representative set under-samples the rare groups where the model is most likely to fail.

How often should an evaluation set be refreshed?

Keep stable versions for comparing models over time, and add a new hidden slice each quarter or on any major change in the input population, so that tuning never overfits the whole set.

Evaluation test set design for AI models: slices, size, leakage