Test Automation
What to automate, in which layer, with what kind of tool, and how to keep the suite trustworthy once it exists. Six guides on the engineering side of testing, written to stay true as products come and go.
6 guidesUpdated Editorial method
About test automation
Most automation programmes fail in one of three ways: they automate the wrong things (the manual regression checklist, verbatim, at the UI layer), they pick tools by fashion rather than fit, or they let flakiness erode trust until nobody reads the results. The six guides here address those failures in order. None of them names a product. Tool landscapes turn over every few years; the reasoning about layers, scope, ownership and reliability does not, and it is the reasoning that a guide should carry.
Test automation strategy is the starting point: a candidate-scoring table (frequency, stability, risk, manual cost) that decides what to automate, ownership models, environments and data, and a twelve-month roadmap by quarter. The test automation pyramid explains the layer ratios and what each layer costs to write, run and maintain, and is clear about when the pyramid is the wrong shape for a system. Open source testing tools goes category by category (unit, API, browser, mobile, performance, security, management, contract) with maturity signals, licence families and a ten-point evaluation checklist; it continues the subject of the magazine's open source tools issue, issue 12.
Flaky tests ranks the causes, gives a rerun-statistics rule for calling a test flaky, and sets out a quarantine policy with time limits; it is the guide to read when the team has stopped trusting a red build. Model-based testing shows how a state model generates test cases and where that pays. Continuous testing maps test kinds onto pipeline stages with time budgets and gates, and argues for a commit stage under ten minutes.
Three situations decide where to start. If an automation plan is due this week, fill in the candidate-scoring table (frequency, stability, risk, manual cost) for the twenty most-run regression cases and let the scores write the first quarter of the roadmap; use the layer cost table to justify why most of them land below the UI. If the build is red and nobody trusts it, go to the rerun-statistics rule for calling a test flaky first, quarantine what fails it with a time limit, and then read the ten-minute commit stage budget, because a suite that takes forty minutes gets skipped and a skipped suite is never trusted again. If an audit is coming, the same pipeline gates and time budgets are what the assessor will map against the release process, the licence-family table in the tool evaluation checklist answers the question legal will ask about what is running in production, and the process side of the audit is TMMi.
The hub connects to the authenticity lane wherever a check has to run on every change. Prompt regression testing is the strategy guide's regression suite applied to a language-model feature, with one difference: the output varies between runs, so the suite passes on rates over repeated runs, and the flaky-test rerun rule gives way to layered assertions and an agreed pass rate. A content provenance pipeline test is a pipeline test in the continuous-testing sense: a fixture of signed images, one stage per gate, a validator as the assertion, run nightly and on every change to the image service. Metadata survival testing is the same job for the EXIF, XMP and C2PA layers, and both belong in the automation roadmap of any team that publishes images. The automation reader should also see what to pin so an AI evaluation reruns to the same rates, the reproducibility question behind every flaky suite, and data lineage for the evaluation sets a pipeline runs on.
The guides cross-reference the coverage and metrics guides in Testing Techniques, because an automation suite is only as good as what you can say about it, and the definition of done guide in Agile Testing, because automation items belong in the DoD or they do not happen.
All guides
Nothing matches. Clear the filter to see every entry.
Questions
What should we automate first?
Score every regression area from 1 to 5 on frequency of execution, stability of the interface, business risk and manual cost, then automate in descending score order. The strategy guide also covers ownership and a first-year roadmap by quarter: foundation, breadth, speed, then pruning.
What ratio of unit, API and UI tests should we aim for?
Roughly 70 percent unit, 20 percent service or API and 10 percent UI tests by count is a starting point, not a rule. UI tests take hours to write and fail for reasons unrelated to the product; invert the shape and the suite becomes slow, brittle and distrusted.
How do I know a test is flaky?
Rerun the failed test against the same commit: 10 reruns with at least one pass and one fail confirms it. Quarantine confirmed flaky tests out of the gating run for at most 14 days while an owner fixes them. Nearly half of flaky tests come down to async timing.
How fast should the pipeline be?
Give each stage a fixed time budget: commit under 10 minutes, build and component under 15, deployed integration and API under 25, acceptance and end-to-end under 30. Each stage gates the next, and the full suite still runs nightly regardless of what the earlier stages selected.
