Software testing metrics: 12 formulas and what each hides

Metrics covered

12, with formulas

Recommended set

5 on one page

Healthy defect removal efficiency

85 to 95 percent

Healthy leakage

under 5 percent

Flaky rate ceiling

under 2 percent of runs

Goodhart's law

a measure that becomes a target stops measuring

Twelve metrics with formulas

The table gives each metric a formula, a healthy range where one exists, and the thing it hides, because the hidden half is what gets teams into trouble. Ranges are typical for enterprise and product software; embedded and safety-critical work runs tighter. Terms are defined in the glossary.

What each metric hides, in more detail

Three of these deserve a longer look because they are the most misused. Defect density is compared across modules to find the risky ones, which works only if the modules received equal testing effort. A module with density zero is either perfect or untested, and the metric cannot tell you which; pair it with coverage. Pass rate is the number most often shown to management and the easiest to game. A suite at 88 percent becomes 100 percent when the 12 percent are marked as known issues and excluded. Report pass rate together with the count of skipped and quarantined tests. Defect removal efficiency is the best single indicator of test effectiveness but arrives late, because the denominator includes defects found in production over the following months. Use it to judge the process across releases, not to judge a release on its ship date.

Two more hide structural problems. Flaky rate below 2 percent sounds healthy until you learn the pipeline retries every failure three times before recording it; the real rate may be 15 percent. Flaky tests covers how to measure it honestly. Automation coverage is a count of cases and says nothing about the risk they address; a team can reach 80 percent by automating the easy paths and leaving every payment scenario manual.

A one-page dashboard layout

A dashboard that needs scrolling is a report, and reports go unread. Six tiles in two rows fit on a screen and on a printed page. Each tile shows the current value, the trend over the last five releases as a small line, and the target as a thin marker. Color is reserved for the two states that need action: red for outside target, amber for trending toward it.

1. Top left, requirements coverage: the percentage of in-scope requirements with a passing test. Answers whether we tested what we promised.
2. Top middle, execution progress: executed against planned, with the burn-down line. Answers whether we will finish.
3. Top right, pass rate with skipped count: both numbers in the same tile so one cannot hide the other.
4. Bottom left, open defects by severity: S1 and S2 counts and their age. Answers whether we can ship.
5. Bottom middle, defect leakage from the last release: the lagging indicator that keeps the team honest about the previous number.
6. Bottom right, flaky rate or automation share: pick whichever is the current improvement goal, and rotate it when the goal changes.

Goodhart's law and how metrics get gamed

Goodhart's law, as usually stated, says that when a measure becomes a target it ceases to be a good measure. Testing metrics are unusually vulnerable because every one of them can be moved without changing the software. Set a target of 95 percent pass rate and failing tests get deleted. Set a target of 500 test cases and one test becomes 20. Set a target of zero open S1 defects and S1 defects get reclassified as S2 on a Friday afternoon. Set a target of defects found per tester and testers log duplicates. None of this is dishonesty; it is people responding to what they are measured on. The defenses are structural: pair every metric with the one that exposes its gaming (pass rate with skipped count, defect count with severity mix, coverage with defect density), report trends rather than thresholds, and never tie individual performance reviews to any of them.

How to pick five

Start from the decisions the metrics must support, not from what the tool exports. A release meeting needs to decide whether to ship, so it needs coverage, open severe defects with age, and leakage from last time. A process improvement effort needs to know whether testing is finding defects early, so it needs removal efficiency and mean time to detect. A pipeline health review needs flaky rate and execution time. Choose the five that answer the questions your team asks every week, agree the formulas in writing including what counts as a defect and a test case, and revisit the set every two or three releases. A metric that has not changed a decision in three releases should be dropped, however easy it is to collect. The return on testing investment can be argued from any of these, but only when the formulas were fixed before the numbers were collected.

This guide is part of the software testing techniques hub. It is best read alongside test coverage and performance testing, which cover the neighbouring questions.

Work it out