Continuous Integration – a new hope?

To say we’ve customised CruiseControl for our continuous integration process is an understatement. And to say that keeping our CruiseControl scheme running smoothly is a tiring endeavor is a major understatement. To be fair, CC enabled us to grow as we needed, so we kept tweaking and adding new stuff. However, as with any software development process, we’ve reached a point where it is healthy to look at the broad picture and refactor our continuous integration process.

CruiseControl is a 1st generation continuous integration tool. It is not bad, but there is newer stuff around. So, after a lunch with the build doctor I’ve started to investigate some 2nd generation tools, mainly TeamCity and Hudson, in search for the one that suits us.

We certainly customised CC quite a bit in order to get there, so our needs might be tricky to satisfy. But our hope is that next generation tools could provide a better/easier way of replacing our customisations with something more maintainable.

Our setup

We have 3 main projects, all having similar requirements from the continuous integration process. Our build pipeline consists of a ‘master’ build (executing unit tests, code checks and test coverage) and several ‘functional test’ builds (browser tests on IE6, IE7 and Firefox, REST and calculation tests), where the functional tests only deploy and execute on successful master builds.

Both the master and functional test builds are split into master and slaves machines (e.g. the browser tests on IE6 run concurrently on three slave machines). There is a lot of custom communication and synchronization going on to ensure that builds are triggered at appropriate times and the master and slave are on the same page, and it can be quite a pain to maintain.

TeamCity (http://www.jetbrains.com/teamcity/)

TeamCity is a commercial product with some very interesting features: you can have pre-tested commits and it uses a ‘build grid‘ to run the tests.

The pre-tested commits means, instead of commiting directly to the source control the code goes first to the TeamCity server, which runs a pre-commit test suite. If the tests pass the code is commited to the source control, but if they fail the developer is notified of the failure and the offending code is prevented from reaching the repository. The build grid idea also seems like a natural way of splitting our many builds into specific agents without requiring custom scripting. For a comparison between TeamCity and CruiseControl, look at: http://www.jetbrains.com/teamcity/documentation/featureMatrix.html.

TeamCity definetly seems cool, but there are downsides. It is quite pricy and I am unsure how customisable it is. Don’t get me wrong, we want to move away from our super-custom scripts. But what happens when something goes terribly wrong, or we want to add an unsupported code checker? It may be very easy to do, but I need to do a more in-depth investigation.

Hudson (https://hudson.dev.java.net/)

Hudson is an Open Source project, much like CruiseControl. However, it does have some rather nice features , such as support for distributing builds across machines and mechanisms for pinpointing when a given unit test started to fail. It does also have an extensive list of plugins, and some are immediately useful for us, which could make a switch easier.

So, if it is similar to CC, why make the switch? To me, the best reason would be to more naturally support the distribution of builds (and the whole master/slaves thing). This seems like one of our maintenance bottlenecks, and if the promise of easy configuration is true it could make our build pipeline saner. This could also mean we could have an easier time scaling our build infrastructure. But, as with TeamCity, further investigation is still required.