12 Hours of Tests In 15 Minutes

The Thoughtworks Mingle team explain how they massively sped up their Selenium RC tests. We’ve been struggling with this problem for awhile now, so it’s nice to see someone reporting that it can be done! If we can get to the speedy feedback the Mingle team have achieved, developers should be able to correct defects they’ve introduced much more quickly and therefore substantially increase feature productivity.

The Mingle team did these things:

  1. Break the tests up into smaller, parallelisable suites, each of which tests a group of related features.
  2. Tag the suites with labels that tell them which suites are relevant to which features, so developers can run just the (apparently) relevant suites.
  3. Use the tags to identify a “sensible subset” of tests to run at build time (would love to know how they do this!)
  4. For each supported platform (browser, operating system, and so on), set up eight CI servers to parallelise the build work.
  5. Combine results from the different platforms into a single reported result. (I’m not so sure about this one – why not report on all of them on one big build radiator?)

The post claims a reduction in feedback time from 12 hours to 90 minutes, and a recent post to the CITCON list claims a further reduction to 15 minutes. That’s where we’d like to be, though until now we hadn’t been able to find any real-life examples of someone achieving this.