12 Hours of Tests In 15 Minutes

The Thoughtworks Mingle team explain how they massively sped up their Selenium RC tests. We’ve been struggling with this problem for awhile now, so it’s nice to see someone reporting that it can be done! If we can get to the speedy feedback the Mingle team have achieved, developers should be able to correct defects they’ve introduced much more quickly and therefore substantially increase feature productivity.

The Mingle team did these things:

  1. Break the tests up into smaller, parallelisable suites, each of which tests a group of related features.
  2. Tag the suites with labels that tell them which suites are relevant to which features, so developers can run just the (apparently) relevant suites.
  3. Use the tags to identify a “sensible subset” of tests to run at build time (would love to know how they do this!)
  4. For each supported platform (browser, operating system, and so on), set up eight CI servers to parallelise the build work.
  5. Combine results from the different platforms into a single reported result. (I’m not so sure about this one – why not report on all of them on one big build radiator?)

The post claims a reduction in feedback time from 12 hours to 90 minutes, and a recent post to the CITCON list claims a further reduction to 15 minutes. That’s where we’d like to be, though until now we hadn’t been able to find any real-life examples of someone achieving this.

Smart Collections

Collections are all too often seen as simple bags full of some particular type of objects. These collections are then at the mercy of their clients. They get taken apart, have elements added and removed, iterated over, and just plain trampled upon. They are given no responsibility of their own.

It is time for that to change! Collections are people too!

Collections represent important concepts in a domain – concepts that appear over and over again as you build different features in your application. They can enforce constraints and provide many operations that apply to entire sets, lists, or maps of domain objects. Creating custom collections (they do not even have to be part of the Java Collections framework!) gives a home for these collection manipulation concepts. Without this home, you would have to remember to include, and rebuild, the constraints or operations every time you built a feature that involved the collection.

Today I created IDSNumbers to act as a home for operations that we often perform on sets of IDSNumber, particularly in our overly-complex calculation code. So far it just has a sum method, but as I (hopefully we) find other common operations, it will be picking up more abilities that we can apply across all our applications.

What other collections are there that deserve to become citizens in their own right?

Another House Call from the Build Doctor

Our friend the Build Doctor made another visit before Christmas, and I just found the notes. The ideas he gave us below are already paying dividends – for example, one of our developers used the techniques to quickly build a new and much simpler type of automated integration test for his product. Then one of his colleagues used the new testing mechanism and reported significantly faster development because of it!

Build-Process Smells

Good Practices

  • Investigate flickering builds properly. Don’t just click [Force] – that’s naughty. Stop the Line can help, if your feedback loop is fast enough (anything more than 10 minutes and it’s too frustrating to stop).
  • Use the Debian package to install Hudson. I’ve done this on my own machine and it’s seductively easy to keep up to date. What if a new version is really buggy though? I guess you’d just reinstall the old version using apt-get (there are no complex dependencies or installation steps, I’m pretty sure).
  • Keep all your dependencies in jar files. We have one large framework that we recompile and insert into each project. We should probably make it one (or many) jar files.
  • Use Ivy for dependency management. Maybe I’m dumb but I don’t see the problem this would solve for us yet.
  • Add screenshots on Selenium failures – it will be easy and hugely beneficial. (He was right!)
  • Write your own Ant tasks if you find yourself writing a convoluted sequence of Ant commands. It’s not that hard and is much easier to comprehend.

Capture Screenshots of Selenium Failures

We are constantly fighting a battle with browser-test failures. Our browser tests should be telling us where our application is failing, so we can fix defects quickly and get back to writing more great features – but when you can’t see where an error came from, you can waste hours just trying to reproduce it. The latest weapon we’ve deployed in this battle allows us to capture a screenshot of the browser whenever a test fails.

Unlike other implementations we only want to create a screenshot when a test has failed.

Firstly, our browser tests are written as Java unit tests using the excellent SeleniumRC library. Now capturing a screenshot is really easy as Selenium provides a captureScreenshot() method. The tricky bit is to only capture a screenshot when a test fails. A crude way to achieve this is to wrap every test method with a:

    try { 
      // test here
    } catch (Throwable e) { 
      // capture screenshot here
    }

Instead of doing this we leveraged JUnit’s @RunWith annotation to use a custom runner which catches test failures and trigger the screen capture. To generalise the functionality, our runner will call any method annotated with @AfterFailure.

Here is the source code to our custom runner:

RCRunner.java

import java.util.List;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;

/**
 * JUnit runner which runs all @AfterFailure methods in a test class.
 *
 */
public class RCRunner extends BlockJUnit4ClassRunner {

    /**
     * @param klass Test class
     * @throws InitializationError
     *             if the test class is malformed.
     */
    public RCRunner(Class<?> klass) throws InitializationError {
        super(klass);
    }

    /*
     * Override withAfters() so we can append to the statement which will invoke the test
     * method. We don't override methodBlock() because we wont be able to reference 
     * the target object. 
     */
    @Override
    protected Statement withAfters(FrameworkMethod method, Object target, 
                                   Statement statement) {
        statement = super.withAfters(method, target, statement);
        return withAfterFailures(method, target, statement);
    }

    protected Statement withAfterFailures(FrameworkMethod method, Object target, 
                                          Statement statement) {
        List<FrameworkMethod> failures =
            getTestClass().getAnnotatedMethods(AfterFailure.class);
        return new RunAfterFailures(statement, failures, target);
    }
}

RunAfterFailures.java

import java.util.ArrayList;
import java.util.List;

import org.junit.internal.runners.model.MultipleFailureException;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.Statement;

public class RunAfterFailures extends Statement {

    private final Statement fNext;

    private final Object fTarget;

    private final List<FrameworkMethod> fAfterFailures;
    
    public RunAfterFailures(Statement next, List<FrameworkMethod> afterFailures,
                            Object target) {
        fNext= next;
        fAfterFailures= afterFailures;
        fTarget= target;
    }

    @Override
    public void evaluate() throws Throwable {
        List<Throwable> fErrors = new ArrayList<Throwable>();
        fErrors.clear();
        try {
            fNext.evaluate();
        } catch (Throwable e) {
            fErrors.add(e);
            for (FrameworkMethod each : fAfterFailures) {
                try {
                    each.invokeExplosively(fTarget, e);
                } catch (Throwable e2) {
                    fErrors.add(e2);
                }
            }
        }
        if (fErrors.isEmpty()) {
            return;
        }
        if (fErrors.size() == 1) {
            throw fErrors.get(0);
        }
        throw new MultipleFailureException(fErrors);
    }

}

AfterFailure.java

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * Annotation to mark a method to be called when a test fails.
 *
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface AfterFailure {

}

Here is an example test using the new runner.

MyBrowserTest.java

@RunWith(RCRunner.class)
public class MyBrowserTest {
     
    @Test
    public void myTest() {
        // Test here
    }

    @AfterFailure
    public void captureScreenShotOnFailure(Throwable failure) {
        // Get test method name
        String testMethodName = null;
        for (StackTraceElement stackTrace : failure.getStackTrace()) {
            if (stackTrace.getClassName().equals(this.getClass().getName())) {
                testMethodName = stackTrace.getMethodName();
                break;
            }
        }
        
        selenium.captureScreenshot("/tmp/" + this.getClass().getName() + "."
                                   + testMethodName + ".png");
    }
}

Cool Quotes and Ideas from XP Day

Several of us attended London XP Day last month and learnt lots of nifty things. I kept a list of some of the cool quotes and suggestions I picked up. Apologies if I don’t give attributions for some of these, as I didn’t manage to keep track of most participants’ names despite the nice nametags.

  • Often tasks seem too trivial or boring for pairing, but it’s just these tasks that can lead to careless bugs and specialisation. When you hit such a task, pair for five or ten minutes on it, and break up only if both partners agree the task is not worth pairing on.
  • “A story should be a question, not a solution. Customers tend to give us solutions, but we should make them recast these as problems we could solve in many ways.”
  • “Self-organising means you don’t come back to The Boss when it doesn’t work out.”
  • The Greeks gave us definitions and categories that defined Platonic forms, and we thought people’s brains worked this way for two thousand years. Then in the 1970’s, we worked out that actually human minds take a bunch of exemplars and make a fuzzy category out of them – so you literally can’t define the Platonic house. Of course, this is exactly what agile developers do – hit the problem domain with loads of user stories until we figure out what users actually want – because in fact the requirements don’t exist until we define the concepts with exemplars! [This was
    Keith Braithwaite
    . See The Big Book of Concepts for more.]
  • Intentionally forming a buddy relationship with someone in a totally different part of the software development world – and seeing that person regularly over the course of a year – has unexpected benefits, including recruiting opportunities, ideas for new activities like coding dojos, and a shoulder to cry on. Try it! [Me, inspired by Portia Tung and Duncan Pierce. My buddy is Simon Woolf from Loopo.]
  • The NHS have a manager with 200 direct reports none of whom have ever met him. Try implementing Lean thinking in that environment! [Marc Baker]

SVNKit versus JavaHL

When using the Subclipse plug-in to integrate Subversion and Eclipse, there is a choice of two back-ends: one that uses the native Subversion libraries (JavaHL), and a pure Java implementation (SVNKit). There seems to be a massive memory leak in the SVNKit back-end, and the only available work-around is to switch to the JavaHL back-end in the Window->Preferences->Team->SVN dialog. If the JavaHL back-end is not visible in the “SVN interface” menu, it may need to be installed (available in the same repository from which you installed the Subclipse plugin).

Real Options

I recently attended XP day where I participated in an open space on Real Options.

Real Options is about “deferring decisions to the last responsible moment”. By avoiding early commitments, you gain the flexibility in the choices you have later. Real Options give you more time to gather more information and explore more options until you have to make a decision. Financial options are just like these, except you are buying the right to purchase or sell an asset instead of the ability to make a decision.

Why do we use Real Options and what are they useful for?

  • To deal with uncertainty – This allows you to postpone really important decisions until the last possible moment.
  • To deal with risk – If you take time to observe and “see what happens next”, this allows you to gather more information and make better informed decisions.

A Real Option has:

  1. A value.
  2. An expiry condition that will tell you when you have to make a decision.
  3. A buying cost and an exercising cost. The buying cost is the cost that is payable now to implement a decision in the future at a fixed cost. The exercising cost is the fixed cost.

To make it clear there are two types of options:

  • Real Option – where the value is more than the costs put together.
  • Stupid Option – where the costs are more than the value itself.

Of course value depends on context so we have to re-evaluate our portfolio of options depending on the context.

The Real Options decision process

  • For a given decision, identify the real options available. This is gathering all the information of all the opportunities that you might have.
  • Identify the last responsible point at which a decision can be made i.e. the conditions to be met to make a commitment. Decision time = deadline – option implementation time.
  • The first decision is made before the first option expires. Until that expiry date, find out when you need in order to exercise those options. During this time actively seek information and clarify the options available as well as their value. Look for new options as well.
  • Identify option(s) for each condition case and know ahead of time which option to exercise given a particular condition.
  • Attempt to push back the decision time because the longer the wait the more information you have and hopefully the more options you’ll acquire.
  • Understand that cost optimisation is not the same as revenue optimisation or risk reduction. Sometimes it is worth investing in more than one option even though this may cost slightly more.
  • Then you wait and wait and wait until the moment when you have to make the decision. Then you can finally make the decision with confidence because you reviewed all your options upfront.

Real options are common sense, but common sense is not common practice. Instinct tells us that we want to make decisions as quickly as possible especially when we are under pressure. The same instinct tells us that it is better to make the wrong decision than to make no decision at all. This is because we are afraid of uncertainty – we do not want to look indecisive or stupid! When we are under pressure, real options can help us make logical decisions and to get value from risk and uncertainty.

Here is an example of Real Options applied to Christmas shopping for a book.

Jack wants to buy a book for his grandmother for Christmas. He identified two ways of shopping:

  • Online
  • Offline

For online shopping Jack identified two delivery methods:

  • Free UK delivery i.e. the super saver option. This takes 7 days.
  • Overnight delivery i.e. the JIT (Just-In-Time)option

Here are the real options Jack identified:

  1. Online shopping with super saver delivery. He identified the deadline to be 24th December and the implementation time is length of super saver delivery. Therefore the decision point is 18th December.
  2. Online shopping with over night delivery. For this option the decision point is later i.e. the 24th December. However, the exercising cost is slightly higher i.e. He would pay more to be able to decide later.
  3. Offline shopping with delivery by Royal Mail. With this option the implementation time is much longer because Jack would have to shop for the book and then send it by Royal Mail. The decision point is sooner. In fact, when he worked out the decision point, he identified that that option had expired.
  4. Offline shopping with Jack delivering the book personally. This reduces the implementation time and adds value as Jack’s grand mother would be glad to see him when he delivers the book. It would take a day to deliver which is still faster than delivery by royal mail.

Jack identifies that time is not the only expiry condition. He is wise enough to also take into account the risk of availability. So he identifies the following ways of extending his off line options:

  1. He would look around shops within a two mile radius and then 10 miles just so that he knows what retailers are available and how he can get a hold of this present.
  2. He would look for alternative gift ideas. This would increase the number of real options he has.

Another online option Jack thought of is to buy a gift voucher. He could buy this online on the 23rd December and then take it over to his grandmother on the 24th. This would take the decision point to a day before the deadline date and the decision of selecting which book would be moved to after Christmas. In this case the decision making process would be deferred to his grandmother so that she can decide which book she wants to buy.

To summarise, a real option has a value, it has an expiry condition, a buying cost, an exercise cost. Use the optimal decision process. Don’t decide now, decide when necessary. Keep your options open, gather information and take the decision when you need to, with confidence.