Flaky tests: causes, detection and a quarantine policy

Reruns to confirm

10 against the same commit

Quarantine limit

14 days, then fix or delete

Alarm threshold

flaky rate above 1 percent of suite

Top cause

async waits, about 45 percent

Trust collapse point

false red more often than true red

Cost per false red

20 to 40 minutes of engineer attention

What flaky means and why it spreads

A test is flaky when its outcome varies without a change to the code under test or the test itself. The definition matters because it excludes two things teams often lump in: tests that fail because the environment is genuinely down (an infrastructure incident, not flakiness) and tests that fail because a dependency changed (a real regression, in somebody else's code). Both need fixing, but neither is solved by the patterns below.

Flakiness spreads through behavior, not code. The first flaky test teaches the team to press rerun. After ten, rerun becomes a pipeline step. After fifty, nobody reads a red result until the third attempt, and a real defect ships because it looked like the usual noise. The pipeline is only as valuable as the trust in its red, so the flaky rate is a first-class metric, reported next to pass rate.

Causes ranked, with the fix for each

Detection: how many reruns before you call it flaky

Rerun the failing test, on the same commit and the same agent class, 10 times. If it passes at least once and fails at least once, it is flaky; if it fails all 10, it is a real failure. Ten is enough because a test that flakes half the time has a 1 in 1,024 chance of failing all 10, and a test that flakes only 10 percent of the time will show at least one pass with 99.99 percent probability (the risk in that case runs the other way: it may pass all 10 and hide). For rare flakes, use history instead: a test that has flipped outcome on the same commit at least twice in the last 30 runs is flaky regardless of what a fresh rerun shows.

Automate the classification. The pipeline records every test result against the commit hash; a nightly job marks any test with mixed outcomes on a single commit as flaky and opens a ticket with the last 5 failure logs attached. Flaky rate is then simply the count of tests carrying the mark divided by suite size. Under 1 percent is healthy; above 3 percent, the gating run has stopped meaning anything.

A quarantine policy that does not become a graveyard

Quarantine moves a confirmed flaky test out of the gating run so the team gets a trustworthy red again, while keeping the test running and reporting so it is not forgotten. The policy needs four rules and a limit, or it becomes the place tests go to die.

1. Entry. Only tests classified flaky by the rerun or history rule enter quarantine. A test cannot be quarantined because it is inconvenient.
2. Ownership. Each quarantined test has a named owner from the team that owns the feature it covers, assigned on entry.
3. Visibility. Quarantined tests still run on every pipeline and their results appear on the dashboard in a separate column, so a fix can be verified against 20 green runs.
4. Limit. 14 days. On day 14 the test is either fixed and readmitted after 20 consecutive passes, or deleted with a note explaining what coverage was lost and how it will be replaced.
5. Cap. Quarantine holds at most 2 percent of the suite. Beyond that, new automation work stops until the count is back under the cap.

Fixing patterns in detail

Async waits. Every fixed sleep in a test is a bet that the system will be fast enough. Replace each with a wait that polls a condition (element visible, response received, queue drained) at a short interval up to a single configured timeout, and make the timeout generous (30 seconds is fine) because a condition-based wait returns the moment the condition holds. Where the system offers no observable condition, that is a testability defect to raise with the developers, not a reason for a longer sleep.

Shared state and order. The rule is that any test can run alone, first, last, or in parallel with any other. Enforce it mechanically: shuffle order in the nightly run, and run a random 10 percent of tests in isolation each night. For data, prefer creation through an API with a unique prefix per run over shared fixtures; for in-memory state, reset singletons and caches in teardown. Environment. Treat any test that fails on one agent and passes on another as an infrastructure ticket first. Time and locale. Never call the system clock from code under test without an injection point, and run CI in a fixed time zone with the date frozen in tests that depend on it. A glossary of the terms used here is available if the team needs a shared vocabulary.

The cost of ignoring it

Each false red costs 20 to 40 minutes of engineer attention: reading the failure, deciding it is probably flaky, rerunning, checking again. A suite with 500 tests, a 3 percent flaky rate and 30 runs a day generates on the order of 15 false reds daily, which is 5 to 10 engineer hours per day spent confirming that nothing is wrong. Over a quarter that exceeds the cost of fixing every flaky test twice. The larger cost is the one that does not show on a timesheet: the real regression that shipped because a red run looked routine. Report the flaky rate and the estimated attention cost on the same page as the other test metrics, and the budget to fix it tends to appear.

This guide is part of the test automation hub. It is best read alongside continuous testing and test automation pyramid, which cover the neighbouring questions.