Continuous Integration – a new hope?

To say we’ve customised CruiseControl for our continuous integration process is an understatement. And to say that keeping our CruiseControl scheme running smoothly is a tiring endeavor is a major understatement. To be fair, CC enabled us to grow as we needed, so we kept tweaking and adding new stuff. However, as with any software development process, we’ve reached a point where it is healthy to look at the broad picture and refactor our continuous integration process.

CruiseControl is a 1st generation continuous integration tool. It is not bad, but there is newer stuff around. So, after a lunch with the build doctor I’ve started to investigate some 2nd generation tools, mainly TeamCity and Hudson, in search for the one that suits us.

We certainly customised CC quite a bit in order to get there, so our needs might be tricky to satisfy. But our hope is that next generation tools could provide a better/easier way of replacing our customisations with something more maintainable.

Our setup

We have 3 main projects, all having similar requirements from the continuous integration process. Our build pipeline consists of a ‘master’ build (executing unit tests, code checks and test coverage) and several ‘functional test’ builds (browser tests on IE6, IE7 and Firefox, REST and calculation tests), where the functional tests only deploy and execute on successful master builds.

Both the master and functional test builds are split into master and slaves machines (e.g. the browser tests on IE6 run concurrently on three slave machines). There is a lot of custom communication and synchronization going on to ensure that builds are triggered at appropriate times and the master and slave are on the same page, and it can be quite a pain to maintain.

TeamCity (http://www.jetbrains.com/teamcity/)

TeamCity is a commercial product with some very interesting features: you can have pre-tested commits and it uses a ‘build grid‘ to run the tests.

The pre-tested commits means, instead of commiting directly to the source control the code goes first to the TeamCity server, which runs a pre-commit test suite. If the tests pass the code is commited to the source control, but if they fail the developer is notified of the failure and the offending code is prevented from reaching the repository. The build grid idea also seems like a natural way of splitting our many builds into specific agents without requiring custom scripting. For a comparison between TeamCity and CruiseControl, look at: http://www.jetbrains.com/teamcity/documentation/featureMatrix.html.

TeamCity definetly seems cool, but there are downsides. It is quite pricy and I am unsure how customisable it is. Don’t get me wrong, we want to move away from our super-custom scripts. But what happens when something goes terribly wrong, or we want to add an unsupported code checker? It may be very easy to do, but I need to do a more in-depth investigation.

Hudson (https://hudson.dev.java.net/)

Hudson is an Open Source project, much like CruiseControl. However, it does have some rather nice features , such as support for distributing builds across machines and mechanisms for pinpointing when a given unit test started to fail. It does also have an extensive list of plugins, and some are immediately useful for us, which could make a switch easier.

So, if it is similar to CC, why make the switch? To me, the best reason would be to more naturally support the distribution of builds (and the whole master/slaves thing). This seems like one of our maintenance bottlenecks, and if the promise of easy configuration is true it could make our build pipeline saner. This could also mean we could have an easier time scaling our build infrastructure. But, as with TeamCity, further investigation is still required.

4 thoughts on “Continuous Integration – a new hope?”

  1. The problem with the TeamCity delayed-commit approach (at least when I looked at it), is that you have to use a ‘special’ SCM plugin in your eclipse environment. So you’re no longer using SVN, you’re using it at a distance – so mixing commandline and eclipse svn usage doesn’t fly.

    This feels rather like trying to teach a pig to sing. What you want it to do is to merge the changes into trunk IFF they pass the build. But because branching in SVN is such a big, complex thing, it’s not practical to do. Teamcity is a solution – but it’s a hack around a fundamental SCM deficiency.

    You can acheive the same result if you can change your SCM (stable tested branches) with Hudson + GIT (and probably Hudson + Hg) using a post-build merge+push. But if you’re using TeamCity – you already *have* changed your SCM.

    If you move to a DVCS, you can also perform another trick, which is to turn the commit + test on its head.

    In the ‘traditional’ model, a user pushes changes to a central repo, the build machine picks them up, and tests (possibly merging the result and committing to provide stable ‘trunk’).

    In the DVCS model, the build machine can also poll each developers machine, pulling any locally-committed (but not pushed to the central repository) changes, build them and test them – giving the developer information about failing tests before he’s even completely finished.

    If you’re interested in delayed-commit (which is really a form of conditional merge+commit), you should look at DVCSs.

  2. A couple of comments regarding DVCS and pre-tested commit.

    First of all not everyone will consider to change their SCM to DVCS (in particular to GIT).

    Secondly using of personal builds only (without actual commit part) is very useful too and it does not have any side effects (its just a patch which is send to the server).

Leave a Reply

Your email address will not be published. Required fields are marked *