Flaky tests: causes, detection and quarantine

What flaky means and why it spreads

A test is flaky when its outcome varies without a change to the code under test or the test itself. The definition matters because it excludes two things teams often lump in: tests that fail because the environment is genuinely down (an infrastructure incident, not flakiness) and tests that fail because a dependency changed (a real regression, in somebody else's code). Both need fixing, but neither is solved by the patterns below.

Flakiness spreads through behavior, not code. The first flaky test teaches the team to press rerun. After ten, rerun becomes a pipeline step. After fifty, nobody reads a red result until the third attempt, and a real defect ships because it looked like the usual noise. The pipeline is only as valuable as the trust in its red, so the flaky rate is a first-class metric, reported next to pass rate.

Causes ranked, with the fix for each

Figure 2Distribution of flaky test root causes (typical, from published studies and team audits)

Root cause distribution. Timing dominates in UI and integration suites; order dependence and shared state dominate in unit suites that grew without isolation rules.

Six causes, their share, how each announces itself and the fix that removes it rather than hiding it.
Cause	Share	Typical symptom	Fix pattern
Async and timing waits	about 45 percent	Element not found, stale reference, assertion ran before the response arrived; fails more on slow agents	Replace fixed sleeps with condition-based waits that poll until a state is true, with a single timeout constant. Assert on the event, not the elapsed time.
Shared or leaked state	about 15 percent	Passes alone, fails in the full run; a record from another test appears in results	Each test creates its own data with a unique key and deletes it, or runs in a transaction that rolls back. No test reads a fixture another test wrote.
Test order dependence	about 12 percent	Passes in the usual order, fails when shuffled or run in parallel	Run the suite in random order once a day. Any failure is a hidden dependency; fix it by making setup explicit in the dependent test.
Environment and resources	about 12 percent	Port in use, disk full, DNS slow, one agent always fails	Ephemeral environments per stream, resource checks before the run starts, and agents rebuilt from an image on every job.
Time, date and locale	about 8 percent	Fails at month end, on the last day of daylight saving, or on an agent in a different region	Inject a clock the test controls; freeze time; run with a fixed locale and time zone in CI and assert against it.
Randomness and concurrency	about 8 percent	Fails one run in twenty with no pattern	Seed every random source and log the seed. For concurrency, test the invariant under a deterministic scheduler or many iterations rather than a single lucky pass.

Detection: how many reruns before you call it flaky

Rerun the failing test, on the same commit and the same agent class, 10 times. If it passes at least once and fails at least once, it is flaky; if it fails all 10, it is a real failure. Ten is enough because a test that flakes half the time has a 1 in 1,024 chance of failing all 10, and a test that flakes only 10 percent of the time will show at least one pass with 99.99 percent probability (the risk in that case runs the other way: it may pass all 10 and hide). For rare flakes, use history instead: a test that has flipped outcome on the same commit at least twice in the last 30 runs is flaky regardless of what a fresh rerun shows.

Automate the classification. The pipeline records every test result against the commit hash; a nightly job marks any test with mixed outcomes on a single commit as flaky and opens a ticket with the last 5 failure logs attached. Flaky rate is then simply the count of tests carrying the mark divided by suite size. Under 1 percent is healthy; above 3 percent, the gating run has stopped meaning anything.

A quarantine policy that does not become a graveyard

Quarantine moves a confirmed flaky test out of the gating run so the team gets a trustworthy red again, while keeping the test running and reporting so it is not forgotten. The policy needs four rules and a limit, or it becomes the place tests go to die.

Entry. Only tests classified flaky by the rerun or history rule enter quarantine. A test cannot be quarantined because it is inconvenient.
Ownership. Each quarantined test has a named owner from the team that owns the feature it covers, assigned on entry.
Visibility. Quarantined tests still run on every pipeline and their results appear on the dashboard in a separate column, so a fix can be verified against 20 green runs.
Limit. 14 days. On day 14 the test is either fixed and readmitted after 20 consecutive passes, or deleted with a note explaining what coverage was lost and how it will be replaced.
Cap. Quarantine holds at most 2 percent of the suite. Beyond that, new automation work stops until the count is back under the cap.

Test lead's rule

A quarantine with no exit date is a deletion with extra steps. Publish the list weekly with days remaining per test, and treat day 14 as a hard stop.

Fixing patterns in detail

Async waits. Every fixed sleep in a test is a bet that the system will be fast enough. Replace each with a wait that polls a condition (element visible, response received, queue drained) at a short interval up to a single configured timeout, and make the timeout generous (30 seconds is fine) because a condition-based wait returns the moment the condition holds. Where the system offers no observable condition, that is a testability defect to raise with the developers, not a reason for a longer sleep.

Shared state and order. The rule is that any test can run alone, first, last, or in parallel with any other. Enforce it mechanically: shuffle order in the nightly run, and run a random 10 percent of tests in isolation each night. For data, prefer creation through an API with a unique prefix per run over shared fixtures; for in-memory state, reset singletons and caches in teardown. Environment. Treat any test that fails on one agent and passes on another as an infrastructure ticket first. Time and locale. Never call the system clock from code under test without an injection point, and run CI in a fixed time zone with the date frozen in tests that depend on it. A glossary of the terms used here is available if the team needs a shared vocabulary.

The cost of ignoring it

Each false red costs 20 to 40 minutes of engineer attention: reading the failure, deciding it is probably flaky, rerunning, checking again. A suite with 500 tests, a 3 percent flaky rate and 30 runs a day generates on the order of 15 false reds daily, which is 5 to 10 engineer hours per day spent confirming that nothing is wrong. Over a quarter that exceeds the cost of fixing every flaky test twice. The larger cost is the one that does not show on a timesheet: the real regression that shipped because a red run looked routine. Report the flaky rate and the estimated attention cost on the same page as the other test metrics, and the budget to fix it tends to appear.

Common questions

Should we add automatic retries to the pipeline?

As a temporary measure while quarantine is set up, one retry is defensible. As a permanent step it hides the problem and doubles run time for flaky tests. If you keep a retry, record every test that needed one; that list is your flaky backlog.

How many reruns confirm a test is flaky?

Ten on the same commit. Mixed outcomes confirm flakiness. Ten straight failures mean a real defect. Ten straight passes after one failure mean either a rare flake or a transient environment issue; check history over the last 30 runs before deciding.

Is a flaky test ever acceptable?

In quarantine for up to 14 days, yes. In the gating run, no. A single flaky test in a gating run teaches the team that red might not mean anything, and that lesson is expensive to unlearn.

What flaky rate is normal?

Mature teams hold under 1 percent of the suite. Between 1 and 3 percent is a warning. Above 3 percent, the suite fails for false reasons more often than for real ones on most days, and gating on it is theater.

Who should fix flaky tests, developers or testers?

The team that owns the feature the test covers, with the platform or automation group providing the diagnosis tooling (rerun classification, shuffle runs, failure log capture). Fixing a flaky test usually means changing product code for testability as well as the test, so it needs both roles.

Can we just delete flaky tests?

Sometimes that is right, when the test duplicates coverage that exists lower in the pyramid. Deletion is acceptable at the end of the quarantine window with a written note of the lost coverage. Deleting on the day the test first flakes throws away the signal that something in the system is nondeterministic.

Flaky tests: causes, detection and a quarantine policy