Twelve metrics with formulas
The table gives each metric a formula, a healthy range where one exists, and the thing it hides, because the hidden half is what gets teams into trouble. Ranges are typical for enterprise and product software; embedded and safety-critical work runs tighter. Terms are defined in the glossary.
| Metric | Formula | Typical healthy range | What it hides |
|---|---|---|---|
| Defect density | defects found / size (KLOC, function points or stories) | 0.5 to 2 per KLOC in test; varies widely | How hard anyone looked; an untested module has density zero |
| Defect removal efficiency | defects found before release / (found before + found after) x 100 | 85 to 95 percent | Needs 60 to 90 days after release before it can be computed |
| Test execution rate | tests executed / tests planned x 100 | Tracks to plan; 100 percent by exit | Whether the executed tests were the important ones |
| Pass rate | tests passed / tests executed x 100 | 95 percent or higher at exit | Deleted or skipped failing tests raise it instantly |
| Requirements coverage | requirements with at least one linked test / total requirements x 100 | 100 percent for must-have requirements | One weak test per requirement counts the same as ten strong ones |
| Defect leakage | defects found after release / total defects found x 100 | Under 5 percent | Depends on how well production defects are reported and linked |
| Mean time to detect | average (time defect introduced to time found) | Hours for unit-level, days for system-level | Requires knowing when a defect was introduced, which is often guessed |
| Flaky rate | runs with non-deterministic outcome / total runs x 100 | Under 2 percent | Reruns that mask flakiness before it is counted |
| Automation coverage | automated test cases / total test cases x 100 | 60 to 80 percent for regression | Counts cases, not value; 500 trivial unit tests inflate it |
| Escaped defects per release | production defects attributed to the release, by severity | Zero S1, few S2 per release | Attribution to a release is often disputed |
| Cost per defect | total test effort cost / defects found | Falls early, rises late in a cycle | Rising cost can mean quality improved or testers stopped looking |
| Defect age | average (time found to time closed), by severity | S1 under 1 day, S2 under 5 days | Closed as will-not-fix shortens it without fixing anything |
What each metric hides, in more detail
Three of these deserve a longer look because they are the most misused. Defect density is compared across modules to find the risky ones, which works only if the modules received equal testing effort. A module with density zero is either perfect or untested, and the metric cannot tell you which; pair it with coverage. Pass rate is the number most often shown to management and the easiest to game. A suite at 88 percent becomes 100 percent when the 12 percent are marked as known issues and excluded. Report pass rate together with the count of skipped and quarantined tests. Defect removal efficiency is the best single indicator of test effectiveness but arrives late, because the denominator includes defects found in production over the following months. Use it to judge the process across releases, not to judge a release on its ship date.
Two more hide structural problems. Flaky rate below 2 percent sounds healthy until you learn the pipeline retries every failure three times before recording it; the real rate may be 15 percent. Flaky tests covers how to measure it honestly. Automation coverage is a count of cases and says nothing about the risk they address; a team can reach 80 percent by automating the easy paths and leaving every payment scenario manual.
A one-page dashboard layout
A dashboard that needs scrolling is a report, and reports go unread. Six tiles in two rows fit on a screen and on a printed page. Each tile shows the current value, the trend over the last five releases as a small line, and the target as a thin marker. Color is reserved for the two states that need action: red for outside target, amber for trending toward it.
- Top left, requirements coverage: the percentage of in-scope requirements with a passing test. Answers whether we tested what we promised.
- Top middle, execution progress: executed against planned, with the burn-down line. Answers whether we will finish.
- Top right, pass rate with skipped count: both numbers in the same tile so one cannot hide the other.
- Bottom left, open defects by severity: S1 and S2 counts and their age. Answers whether we can ship.
- Bottom middle, defect leakage from the last release: the lagging indicator that keeps the team honest about the previous number.
- Bottom right, flaky rate or automation share: pick whichever is the current improvement goal, and rotate it when the goal changes.
Goodhart's law and how metrics get gamed
Goodhart's law, as usually stated, says that when a measure becomes a target it ceases to be a good measure. Testing metrics are unusually vulnerable because every one of them can be moved without changing the software. Set a target of 95 percent pass rate and failing tests get deleted. Set a target of 500 test cases and one test becomes 20. Set a target of zero open S1 defects and S1 defects get reclassified as S2 on a Friday afternoon. Set a target of defects found per tester and testers log duplicates. None of this is dishonesty; it is people responding to what they are measured on. The defenses are structural: pair every metric with the one that exposes its gaming (pass rate with skipped count, defect count with severity mix, coverage with defect density), report trends rather than thresholds, and never tie individual performance reviews to any of them.
Never show a metric alone. Pass rate with skipped and quarantined count. Defect density with coverage. Automation share with the list of manual-only risk items. Defect age with the will-not-fix count. A paired metric is hard to game without the pair moving.
How to pick five
Start from the decisions the metrics must support, not from what the tool exports. A release meeting needs to decide whether to ship, so it needs coverage, open severe defects with age, and leakage from last time. A process improvement effort needs to know whether testing is finding defects early, so it needs removal efficiency and mean time to detect. A pipeline health review needs flaky rate and execution time. Choose the five that answer the questions your team asks every week, agree the formulas in writing including what counts as a defect and a test case, and revisit the set every two or three releases. A metric that has not changed a decision in three releases should be dropped, however easy it is to collect. The return on testing investment can be argued from any of these, but only when the formulas were fixed before the numbers were collected.
Common questions
What is the single most useful testing metric?
Defect removal efficiency, because it measures whether testing catches defects before customers do. Its weakness is that it arrives months late, so pair it with leading indicators such as coverage and defect age.
How is defect density different from defect leakage?
Density is defects per unit of size and is used to compare modules. Leakage is the share of defects that escaped to production and is used to judge the test process as a whole.
Should we track test cases written per tester?
No. It rewards splitting tests and penalizes the tester who spends a day on the one exploratory session that finds the release-blocking defect.
How many metrics should a dashboard show?
Five or six. More than that and readers scan for the green ones and stop thinking. Rotate the sixth tile to match the current improvement goal.
How do we measure metrics in an agile team without formal test cases?
Replace test case counts with story-level coverage (stories with passing acceptance tests) and use defect age, leakage and flaky rate unchanged. The formulas survive; only the unit of counting changes.
What is a good automation coverage percentage?
For regression suites, 60 to 80 percent of cases is common, but the better question is what share of high-risk scenarios is automated. A team at 50 percent with every payment path covered is in better shape than one at 85 percent without them.
Is cost per defect a useful metric?
Only as a trend within one project. Across projects it penalizes good code, because finding the few defects in a well-built system costs more per defect than finding many in a poor one.
Sources
- DORA research program, software delivery performance metrics
- ISO/IEC/IEEE 29119-3:2021, Software testing, Part 3: Test documentation
- ISO/IEC 25010:2023, SQuaRE product quality model
Further reading named in the text
- Stephen H. Kan, Metrics and Models in Software Quality Engineering, 2nd edition (Addison-Wesley, 2002)
- Capers Jones, Olivier Bonsignour, The Economics of Software Quality (Addison-Wesley, 2011)
- Rex Black, Managing the Testing Process, 3rd edition (Wiley, 2009)
- Charles Goodhart, Problems of Monetary Management: The UK Experience (Reserve Bank of Australia, 1975)
This guide is part of the software testing techniques hub. It is best read alongside test coverage and performance testing, which cover the neighbouring questions.