Inter-rater agreement for AI evaluation labels

Why raw agreement is not enough

If two annotators label documents as human-written or machine-written and 90 percent of the set is human-written, they will agree on most cases even if one of them is guessing. Chance-corrected statistics remove that effect. Cohen's kappa compares observed agreement between two annotators with the agreement expected from their label frequencies alone; Krippendorff's alpha generalises the idea to more annotators, missing labels and ordered scales. Artstein and Poesio's survey for computational linguistics explains when each fits and warns against reading fixed thresholds too literally.

A labelling process that produces trustworthy data

1. Write guidelines with a definition of each label, five or more worked examples per label, and explicit rules for the hard cases you already know about.
2. Train annotators on a pilot batch and discuss disagreements before the real work begins.
3. Double-label an overlapping sample, at least 10 to 20 percent of the set and never fewer than 200 cases, with annotators working independently.
4. Compute kappa or alpha on the overlap, per label and per slice, not just overall.
5. Where agreement is low, read the disputed cases, revise the guidelines and relabel; low agreement is usually a specification problem.
6. Adjudicate remaining disagreements with a third person and record the reason, so the rule can be added to the guidelines.

This is the same loop testers know from reviewing requirements: ambiguity found early in the specification is cheaper than ambiguity found after the product ships, as the guide to sampling-based inspection shows for documents. For evaluation sets the specification is the annotation guideline.

How much agreement is enough

Worked example: kappa on a 400-case overlap

Two annotators labelled the same 400 documents as human-written or machine-written before and after the guidelines were revised. The counts are invented but realistic, and they are the counts behind the figure at the top of this guide.

The arithmetic for the first round: observed agreement is (264 + 72) / 400, or 0.84. Annotator A called 302 documents human and 98 machine; annotator B called 290 human and 110 machine. Expected agreement by chance is (302 x 290 + 98 x 110) / 400 squared, which is 0.615. Kappa is (0.84 minus 0.615) / (1 minus 0.615), or 0.58. Reading the 64 disputed cases showed that most were essays with a machine-generated outline and human prose, which the guidelines had not addressed. A new rule (label by the majority of the prose, record the outline separately) removed most of the disagreement in the second round, where observed agreement rose to 0.93, expected agreement by chance was 0.64, and kappa reached 0.81. The improvement is entirely in the disputed cells: 64 disagreements fell to 28, and the remaining ones were adjudicated with a written reason each.

Common mistakes when measuring agreement

• Letting annotators confer during the overlap, which measures how well they negotiate rather than how clear the guidelines are.
• Reporting one overall figure. Agreement on the majority label is usually high; the figure that matters is agreement on the minority label and on the hardest slice.
• Treating 0.8 as a universal bar. The survey by Artstein and Poesio warns that the right threshold depends on the task and on what the labels will decide.
• Fixing low agreement by replacing an annotator. If two careful people disagree, the guideline is the problem, and a third careful person will disagree in the same places.
• Double labelling a convenience sample. The overlap must be drawn at random from the whole set, or the agreement figure describes the easy cases and flatters every accuracy number built on it.
• Reporting agreement once and reusing the labels for years. Guidelines and annotators drift; remeasure whenever the set is extended or a new annotator joins, on a fresh random overlap.

How to report agreement

Report the number of double-labelled cases, the statistic and its value, per label and per slice, next to the accuracy figure they underpin, together with the guideline version and the adjudication rule. Keep the disputed cases and the reason each was resolved, since those decisions are the specification of the label. A usable line reads: kappa 0.81 on 400 double-labelled cases drawn at random from 2,400; 0.86 for the human label, 0.74 for the machine label; guideline v2; 28 cases adjudicated by a third annotator. Anyone reading an accuracy claim built on those labels can then see how much the labels can carry. Where labels feed a benchmark or fairness result that may later be cited against a writer, the label record is part of the evidence, and the guide to evidence-grade test records describes how to keep it so that the labels, the guidelines and the agreement figures stay attached to the numbers they support.

Labels for detector evaluation

For AI text detectors the ideal label is provenance, not judgement: a document is machine-written because it was generated in a recorded session, and human-written because it was written under conditions that exclude assistance. Where provenance is available, use it and keep human judgement out of the label. Where it is not, for example for historical documents, double label, measure agreement and record the uncertainty in the datasheet for the evaluation set. Agreement also matters when a model is used as a grader, as the guides to metamorphic testing and prompt regression testing discuss: the grader must agree with people before its scores mean anything.

This guide is part of the testing AI systems hub. It is best read alongside metamorphic testing for language models and evaluation test set design for ai models, which cover the neighbouring questions.