Exploratory testing: how to write charters and run sessions

What exploratory testing is, and what it is not

Exploratory testing is not random clicking, and it is not what you do when there is no time to write test cases. It is a disciplined approach in which the tester designs the next test using what the previous test revealed, with the design, execution and learning happening in the same sitting. Scripted testing separates those activities across days or people. The exploratory approach keeps them in one head, which is why it finds the defects scripts miss: the ones that live in the interaction between features, in odd data, and in the assumptions nobody wrote down. It complements, and does not replace, the automated checks in your automation strategy.

James Bach and Jonathan Bach gave the approach a management shape around 2000 with session-based test management: timeboxed sessions, each with a charter, each producing a report, each debriefed. Elisabeth Hendrickson's Explore It! gave the charter a template and a large catalog of heuristics. Between them, they made exploratory work accountable without making it scripted.

How to run session-based exploratory testing

1. 1

   Write the charter

   Use the template: explore a target, with resources, to discover information. Target is an area or feature. Resources are the tools, data, personas or conditions you will use. Information is the kind of problem you are looking for. One charter fits on one line; if it needs a paragraph, split it. Write 6 to 10 charters per feature before you start so the day has a shape.

2. 2

   Set up the session

   Book 60 to 120 minutes with no meetings, no chat, no other tickets. Prepare the environment, test data and any observation tools before the clock starts, and note how long setup took. Open a session sheet with the charter, your name, the build number and the start time at the top.

3. 3

   Test and take notes as you go

   Follow the charter, but let what you find redirect you within it. Write short timestamped notes: what you tried, what happened, what you expected. Mark bugs, questions and issues with a distinct symbol so they can be pulled out later. When you leave the charter to chase something interesting, note it; if it is a big enough detour it becomes a new charter.

4. 4

   Close the session at the timebox

   Stop when the time is up, even if you are mid-flow. Fill in the session summary: roughly what percentage of the time went to setup, to testing, and to bug investigation; how much of the charter you covered; what you would do with another hour. Log defects properly, with steps, from your notes.

5. 5

   Debrief with a peer for 15 minutes

   A lead or another tester reads the sheet and asks: what did you test, what did you find, what got in your way, what is left, and how do you feel about this area. Debriefs catch weak charters, share knowledge across the team, and are where new charters come from.

6. 6

   Update the charter list and the coverage view

   Mark the charter done, partly done or blocked. Add the new charters the session produced. Keep a simple grid of feature areas against charters run so anyone can see where the team has and has not looked.

7. 7

   Roll up the numbers weekly

   Count sessions run, charters completed, bugs found per session and the time split. Report them alongside the automated results. Two weeks of data is enough to see whether an area is drying up or still yielding defects.

Writing charters: the template and five examples

A good charter is specific enough to know when you are done and loose enough to follow a lead. Weak charters are either too vague (test the checkout) or a test case in disguise (verify that the coupon field rejects expired codes). The examples below follow the explore-with-to-discover template.

Session notes: what to capture

The session sheet is the evidence. Keep it short and structured so it can be read in the debrief in under five minutes. A working structure: charter and build at the top; tester and start time; a running log of tests and observations with timestamps; bugs marked with a symbol and a one-line summary each; issues (things that blocked or slowed you, such as a missing test account); questions for the product owner; and a closing summary with the time split, coverage estimate and suggested next charters. Percentages for setup, testing and bug investigation are estimates, and that is fine; the pattern across sessions is what matters. A healthy session spends about 10 percent on setup, 70 percent testing and 20 percent investigating bugs. When bug investigation climbs past 40 percent for several sessions, the build is not ready for exploration and you should say so.

Heuristics: tours and SFDPOT

Charters tell you where to look; heuristics tell you how. Tours, from James Whittaker's work, are ways to walk through a product with a purpose. The money tour follows the features customers pay for. The landmark tour jumps between the main screens and checks the transitions. The garbage collector tour visits every corner methodically. The back alley tour uses the features nobody uses. The intellectual tour asks the hardest question the product could be asked. Pick one per session and name it in the charter.

SFDPOT is James Bach's checklist of product dimensions, and it works as a prompt when a session goes quiet. In plain words: Structure is what the product is made of (files, components, services). Function is what it does, including the error handling and the administrative features. Data is everything it reads, writes and remembers, including the odd values. Platform is what it depends on: operating systems, browsers, devices, networks, third-party services. Operations is how real people actually use it, including the ways the designers did not intend. Time is every clock: time zones, timeouts, sequences, concurrency and what happens after a year of data. Run down the list and ask, for each letter, what have I not tried.

Metrics and reporting to management

Exploratory testing earns trust when it produces numbers that mean something. Report four: sessions run this week (throughput), charters completed against charters planned (coverage), bugs found per session (yield), and the time split (health of the build). A yield that falls from 3 bugs per session to 0.5 over three weeks, with coverage rising, is evidence the area is stabilizing. A yield that stays high is evidence it is not, however green the pipeline looks. Present the coverage grid next to the code and requirement coverage figures, and be explicit that they measure different things. For the vocabulary, the glossary defines charter, session and debrief in one line each.

Keep the story simple for people outside the team: here is where we looked, here is what we found, here is where we have not looked yet and why. Managers do not need the session sheets. They need to know that the sheets exist and that someone reads them.

A note on provenance: the address this guide now occupies once served a PDF article on exploratory testing hosted by the original magazine site. That article is no longer available here; what follows is a current, independent explanation of the practice rather than a reproduction of it.

This guide is part of the agile testing hub. It is best read alongside role of tester in agile teams and acceptance test driven development, which cover the neighbouring questions.