Testing Experience Issue 23, September 2013

The issue and its product review

The issue. Issue 23 of Testing Experience was published in September 2013. The magazine, a free quarterly in English for professional testers, was issued as a PDF by a Berlin testing consultancy and training company and edited by Jose Diaz; readers registered and read it at no cost. Issue 23 came in the magazine's sixth year, with five issues still to run before the final number in December 2014, and it belongs to the closing era of the run that began in 2012.

What it covered. Issue 23 is known for a lengthy product review: a long, hands-on assessment of a testing product, the kind of piece a trade magazine of the period ran when a tool mattered to its readers. This page does not name the product or its vendor, because the guides on this site do not carry vendor names and because the review's text is not reproduced here. The other articles in the issue are not described. What the review stands for is a way of choosing tools that was normal in 2013 and has since largely given way to something else.

Tool evaluation, 2013 to 2026

Testing practice in 2013. In 2013 a test manager choosing a tool formed a view from three sources: a review in a trade magazine, a vendor demonstration and a proof of concept on a sample of the team's own work. Commercial suites still dominated functional and performance testing in larger organisations, while open source tools, the theme of the magazine's December 2010 issue, had matured enough to be serious candidates for web and unit-level automation. A long review was valuable because it did work most teams could not afford to do themselves: someone with time installed the product, pushed it beyond the demonstration script and wrote down what broke.

How it changed by 2026. How tools are evaluated changed between 2013 and 2026 more than the tools did. In 2013 a review in a trade magazine carried real weight in a buying decision. Since then the evidence has moved to sources a team can check itself: public issue trackers, release histories, contributor graphs and the licence text, which is why this site's evaluation checklist reads those rather than a reviewer's opinion. The Open Source Definition still draws the line between open source and source-available licences, a distinction that has mattered more since 2018 as several widely used projects changed licence. A review is still useful as a starting list; a bake-off on your own suite is the decision.

Where to read about it now

Where to read about it now. The open source testing tools guide replaces the review with a method. It starts with a ten-point evaluation checklist (release cadence, issue health, contributor spread, breaking-change discipline, documentation, language and platform fit, CI integration, extensibility, licence compatibility and exit cost), then explains how to run a bake-off, which puts two finalists against the same subset of your own suite for a fixed period. For a commercial tool the same checklist applies with licence cost in place of licence family. The test automation strategy guide comes earlier in the order of work: it covers the decision to buy, build or adopt, and its candidate scoring table (six areas, four axes, each scored 1 to 5) decides what the tool will be asked to do before any tool is chosen. The software testing metrics guide covers the measures a team can use afterwards to judge whether the change earned its place.

Reading the original. This site does not distribute the original issues. If you hold a copy of issue 23 and want to cite the review, cite it as Testing Experience, issue 23, September 2013, with an Internet Archive capture if you have one. The magazine hub covers the publication as a whole and how its run ended in December 2014.

Then and now

Guides that continue this subject

• Test AutomationOpen source testing tools: categories, licenses and evaluation
• Test AutomationTest automation strategy: what to automate and in what order
• Testing TechniquesSoftware testing metrics: 12 formulas and what each hides

Back to top