Continuous Quality at uswitch

We were glad to get a visit from our friends at uswitch recently, after Hemal’s very successful skillsmatter presentation. It looks like they’re a little way ahead of us in most areas like functional testing, continuous-integration stability, and continuous deployment. Here’s a summary of what they told us about their cool processes and tools.

Evolution of functional tests

Originally uswitch had completely separate QA and development teams that exchanged little except code and bugs. For instance, QA used a testing tool called QTP that developers had never heard of – they certainly couldn’t run the QA scripts.

Their first step to improve co-operation involved developers and QA staff working together to create Selenium tests, one for each type of user and that user’s typical path through the application (what they call a “moneymaking journey”). These were good for starting the co-operation but they found them to be so flaky that they could only run them manually. It was also impossible to read the Selenium test code, even for developers.

They have now switched to Cucumber and Watir. They find that these two work well together and the biggest advantage is that business people can read the tests (they’ve even had product owners print the tests and edit them on paper to show what they want instead). They still find that the tests flicker: Internet Explorer locks up, the number of processes overwhelm the machine, and more. They have made recent progress in stabilising the tests.

They run a small set of these tests, one suite per business area, automatically during continuous integration; it takes fifteen minutes to run all the functional tests for a relatively large business area. They make failures the team’s top priority; they find that they achieve stability for awhile (for instance with an IE-killer application to eliminate hangs) but then the flickers start again, at which point they have to apply more work to increase reliability.

Some of the methods they use to increase stability include using special access methods that avoid cookies, writing certain pages that accept special query strings, and introducing custom test pages to isolate certain features.

Life of a feature

A uswitch feature starts with a business person (there is one per dev team) defining what the system should do and working with developers to write this up in Cucumber’s expressive given-when-then format. Business people check these tests and are free to add more as they think of important cases.

Each defined feature passes through these stages:

  • Backlog
  • Ready
  • In Progress
  • Inventory
  • Done [i.e. released to production]

Every feature passes through these steps independently – so they can release a new feature the moment it’s finished. They can release many times a day if needed.

There is a backlog of at most ten items in the Backlog state; anything more than this is not worth tracking. uswitch don’t use a bugtracking system; if a feature is still valuable later, they won’t forget it. Instead of identifying test cases in a separate document, they just write the tests in Cucumber and can immediately execute them. (It helps that Cucumber will run a test even if it doesn’t compile.)

Each feature in the backlog is releaseable and valuable on its own – one rule of thumb is that they need to be able to send a press release for each one (it has been hard, but worth it, to train themselves to keep to this). The value may be technical; for instance, they built a new look for some pages, turned it on in production only for internal users until it was ready, then turned it on atomically for everyone. Occasionally they really do need to build a collection of related features that don’t make sense independently, and so they switch temporarily to a standard agile iteration workflow for just that set of features.

Roles and Staffing

uswitch have around 15 developers and 4 QAs. These are divided into four teams, each of which has a business person attached. QAs aren’t really distinguished from devs – they are just devs with a testing focus.

When the business person is away, the team work on tasks that are in the Ready state when they are confident this won’t lead to blockage; if there are none of these, they work on infrastructure tasks or other blockage-reducing activities. They don’t try to develop features without business advice – they find this leads to blockage and rework.

Testers do not act as intermediary between developers and business people. They find a side benefit of the close working is that business people improve their domain expertise.