Continuous testing: which tests run where in the pipeline

The shape of a continuous testing pipeline

A pipeline is a series of gates. Each stage runs the tests that its environment can support, within a budget, and either promotes the build to the next stage or stops it with a specific failure. The earliest stage runs against source and compiled artifacts only and needs no deployed environment, so it is fast and cheap. Later stages deploy the artifact to progressively more production-like environments and run tests that need them. The rule that makes it work: a test runs at the earliest stage where its dependencies exist. A test that could run on the commit but runs nightly is wasting between 8 and 20 hours of feedback latency on every change.

The budgets are targets you defend, not measurements you report. When a stage exceeds its budget for three consecutive days, the team either parallelizes, selects, or moves tests down a layer. Letting the budget drift is how a 10 minute commit stage becomes a 45 minute one that developers stop waiting for.

Keeping the commit stage under 10 minutes

The commit stage is the one developers wait for, so its budget is the strictest. Ten minutes is the figure most teams converge on: long enough for a few thousand unit tests and static analysis, short enough that a developer stays in context. Four techniques hold it there.

• Test only what changed, on the commit. Map tests to the modules they exercise (most build tools produce this from coverage data) and run only the tests whose modules changed, plus their dependents. Typical reduction: 60 to 90 percent of unit tests skipped on an average change. The full set runs nightly to catch mapping errors.
• Split by duration, not by directory. Record how long each test takes and pack tests into parallel streams so every stream finishes at the same time. Four evenly packed streams beat eight uneven ones.
• Cache everything that does not change. Dependency downloads, compiled intermediates and container layers. A cold build that takes 6 minutes should take 40 seconds warm.
• Push slow tests down the pipeline, not into the nightly. A unit test that takes 20 seconds because it touches a database is a component test in disguise; move it to the build stage where the database exists anyway.

Measure the commit stage at the 95th percentile, not the mean. A mean of 7 minutes with a p95 of 22 means one build in twenty makes someone wait 22 minutes, and that is the one they remember.

Parallelization and test selection

Parallelization multiplies environments; test selection divides work. Use both, in that order of preference for the later stages and the reverse for the commit stage. For API and end-to-end tests, start with 4 parallel streams, each with an isolated environment or an isolated data partition, and scale to 8 or 16 when the budget demands. Every parallel stream is an opportunity for order dependence and shared state to surface as flakiness; the shuffled nightly run exists to find those before they reach the daytime pipeline.

Selection for later stages is coarser than for unit tests. Map end-to-end journeys to the services they cross and run a journey when any of its services changed. On a change to one of twelve services, that typically runs a third of the journeys. Run every journey when a shared component (authentication, the API gateway, the design system) changes, and always on the release candidate.

Gates and who can override them

A gate that anyone can skip is a suggestion. Write the override rules into the pipeline configuration, not into a wiki. The commit and build stages have no override: a failing unit test or a broken contract is fixed or reverted, never bypassed. The deploy-to-test stage may be overridden by the owner of the failing service, with a reason recorded in the pipeline log and an automatic expiry after 24 hours, after which the override lapses and the gate closes again. The acceptance gate needs two people, the release owner and the test lead, and the same logging. Overrides are counted and reported weekly; more than two a month on the same gate means the gate is testing the wrong thing or the tests behind it are flaky, and either way it is a backlog item.

Nightly and weekly suites

The daytime pipeline is optimized for speed and runs a selected, parallelized subset. The nightly run is optimized for completeness: everything, unselected, in shuffled order, on a fresh environment, plus the tests too slow for any daytime stage (long random walks, a full performance run against the production-like environment, the complete security scan). Weekly, add the full browser and device matrix and any test that needs a dataset too large to provision nightly. Nightly failures open tickets with logs attached and appear on the team dashboard in the morning; they do not block the daytime pipeline, because a failure discovered at 03:00 should be triaged by a person at 09:00, not by a gate. Define what a green nightly means in the definition of done for the sprint, so it cannot be red for a week without anyone owning it.

Metrics to watch

Four numbers tell you whether continuous testing is working. Lead time from commit to production candidate: the sum of stage durations plus queue time; target under 60 minutes. Stage duration at p95, per stage, against budget. Red-to-green time: how long the main pipeline stays broken once it fails; under 60 minutes is healthy, and a pipeline red for a day is a team that has stopped practicing continuous integration. Escape rate: defects found in production that a pipeline stage should have caught, tagged with the stage that missed them. Put all four on the same page as the other test metrics, review them in the retrospective, and let the escape tags drive where the next tests are written.

This guide is part of the test automation hub. It is best read alongside flaky tests and test automation pyramid, which cover the neighbouring questions.