Telling Stories Around The Codebase

This is a story that has been about a year in the making. It has now reached a point where I think the story needs to be told to a wider audience.

At last year’s Citcon in Paris there was a presentation by Andy Palmer and Antony Marcano. They showed off a way of writing tests in Fitnesse that read much more nicely than you normally encounter. The underlying system they called JNarrate allowed writing tests in Java in a similarly readable fashion.

At youDevise we had been trying to figure out what framework we should be using to try to make things easier to understand and nicer to write. We had taken a look at Cucumber as well as a few other frameworks but none had really made us take the plunge yet. During the conference Steve Freeman mentioned that you don’t need to find a framework. You can instead evolve one by constantly refactoring, removing duplication, and making the code expressive.

Back at work I decided to try to give it a go. Guided by the small bit of JNarrate that I had seen at the conference I started trying to figure out what it would look like. The first passes at it (which got very quickly superseded) tried to do away with some of the things in JNarrate that didn’t seem very necessary. It turned out that many of them were. Not for the code to run, but for the code to be readable.

Eventually it turned into something that looked good.



    .should(be(money(300, usd)));

We worked with this for a while and had very few changes to the basic structure of:

Given.the( <actor> ).wasAbleTo( <action> );
When.the( <actor> ).attemptsTo( <action> );
Then.the( <actor>).exptectsThat( <thing> )
    .should( <hamcrest matcher> );

The framework itself is really just a few interfaces (Action, Actor, and Extractor) with the Given, When and Then classes being the only part of the framework that actually does anything. The rest of what is written is entirely part of the implementing application.

This style of writing non-unit tests has come to pervade large portions of our tests. It immediately helped immensely in communicating with business people and helping developers to understand the domain better. Since it was in Java we had the full support of the IDE for writing the tests and for refactoring them as our understanding of the scenarios improved. Once you get over the initial hurdle of defining the vocabulary, writing up new scenarios becomes so easy that we have started to sometimes go a little overboard with them 🙂

The only change that has occurred recently is that we dropped the standard Java camel-casing of identifiers and replaced them with underscores. We reached this decision after discovering that most of the pain of reading some of our more complex scenarios was in trying to parse the identifiers into sentences. SomeTimesItJustGetsALittleTooHardToFigureOutWhereTheIndividualWordsAre.

So a recent example is:

Given.the( company_admin)
.was_able_to( modify_the_portfolio( the_portfolio)
    .to_add( author_in_other_company_who_published));

Given.the( author_in_other_company_who_published)
.was_able_to( publish_an_idea()
    .with( a_long_recommendation_for( any_stock()))
    .to( a_portfolio_named( the_portfolio)));

Given.the( company_admin)
.was_able_to( modify_the_portfolio( the_portfolio)
    .to_remove( author_in_other_company_who_published));

When.the( company_admin).attempts_to(view_the_available_authors());

Then.the( company_admin)
.should( have( author_in_other_company_who_published));

Who will test the tests themselves?

A short while ago, a colleague and I were developing some JUnit tests using the jMock library, and came across some troubles while trying to start with a failing test. If you’re unfamiliar with jMock, the basic structure of a test looks something like this:

public void theCollaboratorIsToldToPerformATask() {
  // setup your mock object
  Collaborator collaborator = context.mock(Collaborator.class);

  // define your expectations
  context.checking(new Expectations() {{
    oneOf(collaborator).performATask(); // the method 'performATask' should be invoked once

  // set up your object under test, injecting the mock collaborator
  MyObject underTest = new MyObject(collaborator);

  // execute your object under test, which should at some point, invoke collaborator.performATask()

  // check that the collaborator has been called as expected

(For an excellent background on developing software with this technique, I highly recommend reading Growing Object-Oriented Software.)

So back to our problem. We couldn’t work out why our unit test, despite the functionality not existing yet, was passing. It didn’t take long for someone with a bit more experience using jMock to point out our error: we were not verifying that the mock was called as expected. In the code above, this translates to: we were missing the call to “context.assertIsSatisfied()“. Since the mock object wasn’t asked if it had received the message, it didn’t have a chance to complain that, no, it hadn’t.

Granted, myself and my pairing partner were not too familiar with jMock, but it seemed like an easy mistake to make, and it got me thinking.

  • How many other developers didn’t realise the necessity to verify the interaction?
  • How many tests had been written which did not start out failing for the right reason, and thus, were now passing under false pretences?
  • How could we check our existing tests for this bug, and ensure that new tests didn’t fall prey to the same lack of understanding?

In short, who will test the tests themselves?

A satisfactory answer for this case, I found, is FindBugs.

FindBugs is a static analysis tool for Java, which detects likely programming errors in your code. The standalone FindBugs distribution can detect around 300 different types of programming errors, from boolean “typos” (e.g. using & instead of &&) to misuse of common APIs (e.g. calling myString.substring() and ignoring the result). Obviously the FindBugs tool can’t anticipate everything, and the jMock error was obscure enough that I had no expectation of it being included. Fortunately, a handy feature of FindBugs is that, if you have a rough idea of a bug you’d like to discover, and a couple of hours to spare, you can write your own plugin to detect it.

With a bit of effort I had whipped together a simple detector which would find this type of problem across the host of tests we continuously run at youDevise. Out of approximately 4000 unit tests, this error appeared around 80 times. Not too many, but enough to be concerned about. Fortunately most of the time, when the call to context.assertIsSatisfied() was included (or the @RunWith(JMock.class) annotation added to the class), the tests still passed. That they “fortunately” still passed, was the problem, since that depended on luck. Occasionally the problem test cases didn’t pass after being fixed, and it either meant a test was outdated and the interaction deliberately didn’t happen anymore, or the interaction had never occurred in any version of the code. Fortunately (again, more by luck than judgment) the tests didn’t actually highlight faulty code. Granted, we also have suites of higher level tests, so the unit tests were not the last line of defense, but still, it is important that they do their job: providing fast (and accurate) feedback about changes, and communicating intent.

The FindBugs plugin, by testing the tests, helped to discover when they weren’t doing their job. Since we run FindBugs, now with said plugin, as part of the continuous build, it will (and has) prevented new test code from exhibiting the same fault. Although no bugs in production code were revealed as part of correcting the tests, knowing that we won’t need the same flavour of luck again increases confidence. This in turn leads to all manner of warm and fuzzy feelings.

Since the plugin has been open-sourced, if you or your team uses jMock, you can detect and prevent those errors too (though I don’t guarantee you’ll feel as warm and fuzzy as I did).

The FindBugs plugin is available for use from the youDevise github repository. Instructions and the JAR to download are included. FindBugs is also open-source, is free to use, mature, and widely used (by the likes of those reputable sorts at Google) and is available for download.

So if you ever find yourself asking,“Who will test the tests themselves?” maybe FindBugs, with a custom plugin, is your answer.