Variants of Session-Based Testing

Antony Marcano has been telling us for awhile about the wonders of Session-Based Testing, an idea suggested by Jon and James Bach. It’s a way to record, manage, and audit what clever exploratory testers already do – verify features and find bugs using a rough description of the area to be tested, and strike out in interesting directons if a promising avenue for testing opens up.

We decided finally to try out Session-Based Testing on our HIP project, but with some modifications, and the results so far are very positive – more smiling faces during and after testing, and continued high quality. Here’s a comparison of our process to the standard one described by Antony and the Bachs:

Charter Generation
Antony suggests generating charters before coding (as part of acceptance-test-driven development [PowerPoint]) and refining the charters after coding. It’s also helpful to generate the charters early so you can include the time in iteration planning. He finds that brainstorming sessions tend to last around 30 minutes or so per feature.

The Bachs are pure testers, so generate their charters after development. They don’t say much about how they brainstorm the charters.

We generate charters after completing code for an iteration. We hold one big brainstorming session of about an hour, covering all the features for that iteration (typically 2-4 major features). We include a full day of testing in every iteration plan.

Antony recommends sessions of 25-45 minutes if alone, perhaps up to an hour if pairing. Where possible, it’s helpful to use successful charters as the basis of new automated tests – for instance, if you have a charter for “Entering invalid customer orders”, you might want to create a suite of integration tests that exercise your validation code by entering some of the invalid order types you used during the session.

The Bachs run single-tester sessions of “around 90 minutes”, though it sounds like they may take much longer or much shorter in some cases. They don’t mention test automation. Recorded charters seem to be used mostly for auditing by their customer.

We pair on all our sessions and they last 1-2 hours (we typically do three in one day, similar to the Bachs). We record all our charters on a wiki. We haven’t yet built automated tests corresponding to our successful charters, but hope to do so.