Test Automation Guides
What to automate, in which layer, with what kind of tool, and how to keep the suite trustworthy once it exists. Six guides on the engineering side of testing, written to stay true as products come and go.
Most automation programmes fail in one of three ways: they automate the wrong things (the manual regression checklist, verbatim, at the UI layer), they pick tools by fashion rather than fit, or they let flakiness erode trust until nobody reads the results. The six guides here address those failures in order. None of them names a product. Tool landscapes turn over every few years; the reasoning about layers, scope, ownership and reliability does not, and it is the reasoning that a guide should carry.
Test automation strategy is the starting point: a candidate-scoring table (frequency, stability, risk, manual cost) that decides what to automate, ownership models, environments and data, and a twelve-month roadmap by quarter. The test automation pyramid explains the layer ratios and what each layer costs to write, run and maintain, and is clear about when the pyramid is the wrong shape for a system. Open source testing tools goes category by category (unit, API, browser, mobile, performance, security, management, contract) with maturity signals, licence families and a ten-point evaluation checklist; it continues the theme the magazine's most linked issue, issue 12, was cited for.
All 6 guides in this hub
| # | Guide | What it settles | Level | Reading |
|---|---|---|---|---|
| 1 | Test automation strategy: what to automate and in what order | A test automation strategy is three decisions written down: what gets automated first, who owns the code that does it, and what it costs to keep running. | Lead | 8 min |
| 2 | Test automation pyramid: ratios, costs and when to break it | The test automation pyramid is a budgeting device for automated checks. | Practitioner | 7 min |
| 3 | Open source testing tools: categories, licenses and evaluation | Nearly every test team runs on open source somewhere: the unit runner, the browser driver, the load generator, the proxy. | Practitioner | 8 min |
| 4 | Flaky tests: causes, detection and a quarantine policy | A flaky test passes and fails against the same code. One in a suite is a nuisance. | Practitioner | 8 min |
| 5 | Model based testing: models, generation strategies, coverage | Model based testing generates test cases from a model of how the system should behave instead of writing each case by hand. | Practitioner | 8 min |
| 6 | Continuous testing: which tests run where in the pipeline | Continuous testing means every change is tested at every stage it passes through, automatically, inside a time budget the team can wait for. | Lead | 8 min |
Where to start, by the question you have
Flaky tests ranks the causes, gives a rerun-statistics rule for calling a test flaky, and sets out a quarantine policy with time limits; it is the guide to read when the team has stopped trusting a red build. Model-based testing shows how a state model generates test cases and where that pays. Continuous testing maps test kinds onto pipeline stages with time budgets and gates, and argues for a commit stage under ten minutes.
Three situations decide where to start. If an automation plan is due this week, fill in the candidate-scoring table (frequency, stability, risk, manual cost) for the twenty most-run regression cases and let the scores write the first quarter of the roadmap; use the layer cost table to justify why most of them land below the UI. If the build is red and nobody trusts it, go to the rerun-statistics rule for calling a test flaky first, quarantine what fails it with a time limit, and then read the ten-minute commit stage budget, because a suite that takes forty minutes gets skipped and a skipped suite is never trusted again. If an audit is coming, the same pipeline gates and time budgets are what the assessor will map against the release process, the licence-family table in the tool evaluation checklist answers the question legal will ask about what is running in production, and the process side of the audit is TMMi.
The hub connects to the authenticity lane wherever a check has to run on every change. Prompt regression testing is the strategy guide's regression suite applied to a language-model feature, with one difference: the output varies between runs, so the suite passes on rates over repeated runs, and the flaky-test rerun rule gives way to layered assertions and an agreed pass rate. A content provenance pipeline test is a pipeline test in the continuous-testing sense: a fixture of signed images, one stage per gate, a validator as the assertion, run nightly and on every change to the image service. Metadata survival testing is the same job for the EXIF, XMP and C2PA layers, and both belong in the automation roadmap of any team that publishes images. The automation reader should also see what to pin so an AI evaluation reruns to the same rates, the reproducibility question behind every flaky suite, and data lineage for the evaluation sets a pipeline runs on.
The guides cross-reference the coverage and metrics guides in Testing Techniques, because an automation suite is only as good as what you can say about it, and the definition of done guide in Agile Testing, because automation items belong in the DoD or they do not happen.