Agile inspection: sampling requirements for major defects

Why sample instead of reading everything

The purpose of inspection, in Tom Gilb's framing, is not to clean up a document. It is to measure the document's quality and decide whether it is fit for the next step. A full inspection of a 40 page specification takes each reviewer a day of reading, a half-day meeting, and produces a list of hundreds of items that nobody fixes because the deadline has arrived. A sample of two pages, checked properly, tells you the defect density within a useful margin in one hour. If the density is 20 majors per page, you do not need the other 38 pages to know the document must be rewritten; you need to send it back with the numbers and the rule violations that explain them.

Sampling also changes the behavior of authors. When they know two random pages will be checked against rules and the result will be a number, they write to the rules. Density on second and third submissions typically drops by a factor of five to ten. The inspection becomes a quality control gate on the process, not an editing service for one document.

The rules: what counts as a major defect

A defect is any violation of an agreed rule. A major defect is one that could plausibly cause a wrong implementation, a wrong test, or significant rework downstream. Spelling, formatting and style are minor and are not counted. Keep the rule set short (five to seven rules) so checkers can hold it in their heads; the rules below are the core set most teams start from.

How to run a one-hour agile inspection

1. Kickoff, 10 minutes. The leader states the purpose (measure, not fix), hands out the rules, and picks the sample: one or two pages chosen to be representative, not the introduction and not the appendix. Agree the page size as about 300 words so densities compare across documents.
2. Individual checking, 30 minutes per page. Two or three checkers read the sample alone, slowly, marking every rule violation and rating it major or minor. Slow matters: effective checking rates are around one page per half hour. Faster reading finds a fraction of what is there.
3. Logging, 10 minutes. Checkers report counts of majors, then merge lists to remove duplicates. Do not debate whether each item is really a defect; the author decides later. Record unique majors.
4. Calculate, 5 minutes. Unique majors divided by pages sampled gives the density. Multiply by total pages for the document estimate. Apply a correction for what checkers miss: experienced teams find that even good checkers catch a third to a half of the majors present, so the true density is likely two to three times the observed one.
5. Decide, 5 minutes. Compare against the exit criterion. Below one major per page: proceed. Above: return to the author with the numbers and the marked sample, and re-sample a different page after rewrite.

A worked example with numbers

A 12 page functional specification for a pricing module arrives for sign-off. The test lead runs an agile inspection with three checkers: a developer, a tester and a business analyst who did not write it. Pages 3 and 10 are sampled. After 30 minutes per page the checkers report 5, 6 and 4 majors respectively. Merging removes duplicates and leaves 9 unique majors across 2 pages: a density of 4.5 majors per page. Extrapolated to 12 pages, that is about 54 majors in the document. Correcting for the checkers' catch rate suggests the real figure is between 100 and 160.

The exit criterion was fewer than 1 major per page, so the specification is not fit to code from. Total effort: three checkers for 75 minutes each plus the leader for 60 minutes, about five person-hours, and one hour elapsed. Compare that with the cost of discovering the same 100 ambiguities during development and testing, where the multipliers on defect cost apply. The most common majors in this example were unquantified performance statements, three uses of the word appropriate, and a discount rule that contradicted the boundary given two pages later. Each of those would have become a defect, a test that could not be written, or a three-way argument in a sprint review.

What to do with the result

If the document fails, the author fixes the whole document against the rules, not only the sampled pages, and resubmits. Re-sample two different pages. Most authors reach the exit criterion on the second or third pass, and the rules stick for the next document. If it passes, proceed, and record the density: a run of accepted documents at 0.5 majors per page is a process metric worth reporting alongside your other testing metrics. Either way, the marked sample pages go back to the author; the merged list of majors is more useful as teaching material than as a fix list.

In an agile team the natural place for this is refinement. A story with a two page acceptance specification can be sampled in full by two people in 30 minutes before the three amigos session, and the rule violations become the first questions asked in that session (see acceptance test driven development). The technique also works on test plans, architecture documents and contracts; anything with rules can be inspected against them.

On provenance: the address this guide occupies once served a PDF on agile inspection hosted by the original magazine site. That document is no longer available here, and the explanation above is a current, independent account of the technique rather than a copy of it.

This guide is part of the agile testing hub. It is best read alongside acceptance test driven development and agile program management, which cover the neighbouring questions.