Confidence intervals for classifier metrics: a worked method

Why the plus-or-minus formula fails for small rates

The interval most people learn is the rate plus or minus 1.96 times its standard error. It assumes the sampling distribution is roughly symmetric, which is close enough when the rate is near the middle and the sample is large. Detector testing lives at the other extreme: false positive rates of 0.5 to 3 percent measured on a few hundred to a few thousand documents. There the simple formula can produce a lower bound below zero and intervals that are too narrow.

The Wilson score interval, introduced by Edwin Wilson in 1927 and recommended by the NIST/SEMATECH statistics handbook for proportions, handles this correctly. It is asymmetric where it should be, never leaves the zero to one range, and has close to the coverage it claims even for small samples. Every rate in a detector benchmark should carry one.

Compute and report an interval in five steps

1. 1

   Keep the counts, not just the rate

   Record the number of errors and the number of cases for each slice. A rate without its denominator cannot be checked or combined later.

2. 2

   Choose the confidence level before the run

   95 percent is the convention. Write it into the evaluation plan with the pass criterion, as you would any test metric threshold.

3. 3

   Compute the Wilson interval

   Use a statistics library or a spreadsheet formula for the Wilson score interval. Check one value by hand against a published calculator the first time.

4. 4

   Apply the criterion to the upper bound

   If the requirement is a false positive rate below 2 percent, the upper end of the interval must be below 2 percent. A point estimate of 1.4 percent with an upper bound of 2.6 percent does not pass.

5. 5

   Report count, total, rate and interval together

   Write 20 of 1,000 (2.0 percent, 95 percent interval 1.3 to 3.1). A reader can then see both the evidence and the uncertainty.

How many cases you need

Work backwards from the decision. If you must show a false positive rate below 1 percent and you expect about 0.5 percent, you need enough human documents that the upper bound falls under 1 percent, which is typically a few thousand per slice. When no errors are expected, the rule of three gives a quick check: with zero errors in n cases, the 95 percent upper bound is about 3 divided by n. Zero false positives in 300 documents supports a claim of under 1 percent and nothing stronger. The guide to evaluation test set design turns this into per-slice sizes.

Comparing two detectors or two versions

Overlapping intervals do not prove two rates are equal, and non-overlapping intervals are a conservative test of difference. When both detectors run on the same documents, use a paired comparison: count the documents one flags and the other does not, and test whether those disagreements are balanced. That uses the pairing and needs far fewer cases than comparing two independent rates. Treat a change that the test cannot distinguish from noise as no change, however good the new number looks on a slide.

Worked example: a paired comparison of two versions

A team upgrades its detector and wants to know whether the false positive rate really fell. Both versions score the same 1,000 human documents. The counts below are invented but realistic. Version 1 flags 22 documents (2.2 percent, interval 1.5 to 3.3). Version 2 flags 15 (1.5 percent, interval 0.9 to 2.5). The intervals overlap heavily, which on its own settles nothing, so the analysis moves to the disagreements.

Of the 17 disagreements, 12 favour version 2 and 5 favour version 1. If the versions were equally good the split would be close to even. An exact binomial test on 12 against 5 gives a two-sided p value of about 0.14, so the improvement is plausible and unproven: the team would need roughly twice as many disagreements in the same ratio before treating it as real. Reporting only that 2.2 percent fell to 1.5 percent would overstate what 1,000 documents can show.

Common mistakes with intervals

• Reporting the rate without the count and denominator. A reader cannot recompute or combine 1.2 percent; they can recompute 6 of 500.
• Applying the pass criterion to the point estimate. A ceiling of 2 percent is met when the upper bound is below 2 percent, which needs several times more cases than a point estimate that happens to land under it.
• Treating zero errors as zero risk. Zero in n supports an upper bound near 3 divided by n and nothing stronger; zero in 100 is compatible with a true rate of 3 percent.
• Pooling slices to get a narrow overall interval. The overall figure can pass while a slice of 150 second-language writers is compatible with a rate several times higher.
• Recomputing after adding cases until the interval passes. Sequential peeking inflates the false pass rate; fix the sample size before the run.

How to report a rate

The unit of reporting is the count out of the total, the rate, the interval and the method, for every slice, with the version of the evaluation set the numbers came from. A line such as false positive rate 20 of 1,000 (2.0 percent; 95 percent Wilson interval 1.3 to 3.1), evaluation set v3, detector build 2026.09.2, threshold 0.95 can be checked by anyone with a calculator. Keep the per-document scores and verdicts alongside the summary, so that the same run can be re-analysed at another threshold or another confidence level without re-running the detector. Where a benchmark or fairness result may later be cited in a dispute about a specific writer, the guide to evidence-grade test records describes what else the record must preserve for the number to hold up.

This guide is part of the testing AI systems hub. It is best read alongside ROC curves and thresholds for AI text detectors and evaluation test set design for ai models, which cover the neighbouring questions.