Monitoring Hudson CI – Solved!

In an earlier post on our continuous-integration monitoring, I said we were having some challenges in measuring wait time (from checkin to start of build) and build time (from start of build to published result). I’m pleased to report we’ve solved these now, using the clever Hudson Python API.

Hudson has a remarkably discoverable API – to use it, visit the page that shows the data you want, such as

http://myhudson.example.com/job/MyJob

and just add /api on the end:

http://myhudson.example.com/job/MyJob/api

You get a nice page describing what you can do with the API from this page – and one of the choices is to get Python code from a URL like

http://myhudson.example.com/job/MyJob/api/python

Like JSON, you use eval() to convert the code to a Python dictionary that’s easy to use straightaway – no explicit parsing required!

Our CruiseMissile feedback device is written in Python, so it wasn’t hard to get the data we needed about each job and record it in a database (though we did hit some minor snags). Now we run one SQL query and get wait time, build time, and response time for all our builds. Next job is to actually improve those figures, for instance by adding shiny new servers.

Our Film Debut – CruiseMissile

The Build Doctor graced us with his presence again some time ago and brought his camera to film one of our team members – the big monitor that shows you the status of all our builds! Since we used to use CruiseControl, we jokingly named this our CruiseMissile, and the name has stuck even though we’ve switched (happily, mostly) to Hudson. You can see its cinematic debut if you like that sort of thing.

The Doctor had good suggestions, as always. One of our big problems with our CI grid is the sheer size of the artifacts it pushes around. We want to put these artifacts on a diet, but until we get to do that we find ourselves goofing off while we wait for a zip, and then a copy, and then an unzip, over and over.

Julian says we should consider some clever hardware and software solutions like RAMDisks, TMPFS, or a solid-state disk. When using any of these solutions, we could pass the location of the file as the artifact and never copy it at all. We may also find that uncompressed transmission is faster than compressing, since all the traffic is local.

We aren’t actually sure whether this is the bottleneck now, since we went to a new version of Hudson that copies in a slightly less silly way. One problem is that you can’t add timestamps to the Hudson console (I feel a plugin coming on) so we can’t easily see which bits actually take the longest. The Build Doctor reminds us that at least for the Ant part of our build, we could use the nifty Performance Monitoring task in ant-contrib to measure the time for each task.

Code Kata: Tell Don’t Ask

At youDevise we’ve started to run regular code katas, an idea that started in our dojos. Two of us practise a particular themed refactoring many times for a week, then perform it for everyone else along with a description of the theme. It’s early days, but seems to be working OK.

I started a wiki page on the first theme, Tell Don’t Ask, and encouraged developers to add thoughts and code snippets to it. I was pleased to see how many good ideas we had; for privacy reasons I can’t share the code, but here are some of the highlights from the page on this very useful principle.

Tell Don’t Ask

Using the principle Tell, Don’t Ask means sending messages from one object to another, rather than exposing the state of one object to others who manipulate that state. If you follow this principle:

  • You will reduce coupling among your classes.
  • You should be able to give your classes and methods more business-meaningful names, not just getThis() and getThat(). If you’re not sure what name to give a class, look at its methods and ask yourself what domain entity does those things; if the methods don’t suggest an entity, you may not have a coherent class! Business entities rarely, if ever, get(); they may report() or show() but this implies much more than a naked get().
  • Mocking will be easier and will make more sense. Mocks that are told what to do can easily fake the behaviour, or just record their incoming messages. Mocks that have to pretend that they have an internal state manipulated through invasive methods are harder to write and document behaviour poorly.
  • You will avoid confusing, heavily-coupled “train wreck” code: dog.getHead().getMouth().insertFood(steak) instead of dog.eatFood(steak).

TDA vs. DI

Here are two sequence diagrams that illustrate the main differences and similarities between the Tell Don’t Ask principle and the Dependency Injection principle:

Ask without Telling

Ask without Telling
Telling

Telling

The parts to notice are that when using the tell, don’t ask principle there are going to be more objects interacting with each other. Each one of those classes will, however, be much smaller since each one is concerned with just its own little bit of the overall picture. The classes turn into an orchestration of the algorithm, rather than a procedural listing of the algorithm.

Another side effect of tell, don’t ask is that mocking becomes much more useful. The mocks can now make real assertions about what was supposed to be done and check that the correct interaction occurred. This is because the ideas of the algorithm have been elevated from plain, opaque data into meaningful and separated messages.

The other half of the diagrams is the dependency injection. DI has similar properties to Tell, Don’t Ask in that the user of an object is “telling” it what other objects to use, but differs in where its main focus lies. DI is much more about the lifecycle (creation and destruction) of objects and much less about how they are interacted with.

It all boils down to this: without DI testing is nearly impossible, without Tell, Don’t Ask testing is monolithic. With both DI and Tell, Don’t Ask testing is simplified and well separated.

Reducing Complexity

“Tell don’t ask” is about reducing a particular kind of coupling complexity: operations by other objects on the exposed state of a target object. “Tell don’t ask” solves this by pushing the operation into the target object and hiding the associated state in there as well.

The more objects that can see and act on the states of other objects the more complex your system becomes: the exposed states of an object are multipliers of complexity. Control complexity by limiting the exposed states of objects.

Pure “tell don’t ask”, a void function with no exceptions, gives the caller no possibility of behaving differently in response to the action they’ve triggered, there’s no visible state, no return message, nothing for the caller to act on, and that makes it much easier to reason about the correctness of the caller.

The next step up allows is boolean return value with the caller able to follow two branches. A number return allows many branches, exceptions are also returns that cause branching, and so forth. It’s easier to think about just two branches rather than many branches. It’s much easier to think about no possibility of branching. If changes in the target object’s state are visible then anything with access to that object can change it’s behaviour in reponse to the initial operation, multiplying complexity uncontrollably.

When writing a caller class you want to be able to reason about your class, to assure yourself that it is correct, and to know that others looking at it later will feel sure it’s right. Being exposed to less state in other objects helps keep your object simpler, so you should be trying to expose yourself to the least number of objects and you should want them to have simple interfaces that don’t allow needlessly complex possiblities as a response to their operation. If nothing else hide the state of bad objects in facades to show how it could be done.

A programmer writing a target class that will be called by others should be trying to be a good citizen, should be trying to make it easy for the callers to be simple. Offering wide-open interfaces with access to complex state forces the callers to at least think about the possibility of responding to that complexity, and that makes their lives harder: general is not simple. Try to limit the names of public functions to clear business operations – being so specific can help keep the system as simple as possible.

Pairing uses more of your brain

I have a theory that pair-programming is an acceptable alternative to talking to yourself.

I’ve come to this conclusion while working in pairs with people in the team for the greater part of the last few months. I was wondering why, on a purely qualitative level, our team seems to get more out of its developers when they work as half of a pair, compared to when they work solo. We feel that we can work more carefully and more thoroughly in pairs, and we’ve noticed that our pairing days feel more intense. We’re usually more tired at the end of them, than on days when we’ve been working by ourselves.

I guess the basis of my theory is that we inevitably use more of our brains when we pair. It would take some MRI equipment to be truly scientific in verifying this. Probably beyond our development budget. So instead I’ll just talk about why this might be.

Suppose I’m sitting by myself in my study at home, quietly writing some code. If I were a neuroscientist, I would list the bits of my brain I’m using for this task, including parts of the cerebral cortex for expressing my thoughts in the language of the software, the temporal lobes for memory, the parietal lobes for formulating mental pictures and nonverbal ideas, and coordinating hand-eye movements, and so on.

As time passes, I get deeply involved with what I’m doing. If I had a tape recorder in the room I would notice an amusing change in my behaviour. I start talking to myself.

I’m confident this is no reason suddenly to go check myself into the nearest mental health facility. In fact many people do this, if only for a moment, before they look round the office and remember they’re not actually by themselves.

If I start talking to myself while coding, it shows my brain has started to engage other linguistic and speech-related parts of itself, namely the Wernicke and Broca areas in the left hemisphere. It’s almost as if my brain is chipping in more resources to help out with my coding task. And if I’m totally honest, talking aloud like this does help. I seem to be able to think more clearly about a problem if I can start talking about it, if only to the walls.

Of course, these days I do all my real work in an office. It’s an open plan office in which I’m surrounded by people. I’m not about to start talking to myself. I would be an awkward distraction, and everyone would think I was a loon. So when I’m working solo in the office, Broca and Wernicke can’t get involved, and I have to use less of my brain.

Except, if I’m pair programming, I need those linguistic and speech areas, so I can talk about what I’m doing, and listen to the inflow of ideas from my pairer. I can’t pair without them.

It would be wrong to consider the brain as one big fatty coding engine, that the more of it you could deploy for software development, the faster you could churn out product. But I do think involving more of the brain by introducing a spoken interactive element is a big improvement on working in silence in isolation. Especially as ours is a typical open plan office where there is no silence. If coders can’t have silence, better the primary noises going into their head are their pairer talking about what to type next.

One accusation often levelled at pair programming is that it wastes programmer resources: surely the team could complete more tasks in a development cycle if they paired less? I would love to try and measure whether or not this is true. Without a measurement, developers resort to the usual defensive arguments that pairing improves code quality, distributes knowledge over the team, and so on. Meanwhile, I’m happy in the opinion that if need to get more work done, pairing seems to be a good way of getting the most out of my brain while not sounding like a crackpot.

Fall In FIXNOWs

Agile testing expert Antony Marcano has been visiting us periodically for more than a year now. In that time we’ve taken one product from releasing every three months to releasing once a fortnight, introduced a formal quality process, and brought in a full-time QA expert. And most importantly, our level of product quality has shot up. I’ll let Antony compare the level of quality we have now with that of other customers he consults with regularly:

… A handful of FIX-NOW bugs in production in six months is a phenomenally low figure based on my personal experience of some companies. It’s not that uncommon to see one or two fix-now issues per week in production for several weeks after each (infrequent) release (before I’ve had a chance to work with them for a while)… I’ve seen this consume a huge proportion of developers’ time fixing production issues rather than working on new features.

Everyone at youDevise participated in making this change happen, especially our QA specialists (thanks Sai, Jonny, and Wendy!) Antony gets his share of the credit for patiently helping us focus on professionalising our completely informal testing process and helping us begin to rein in and manage our automated tests, especially those pesky slow and inscrutable functional tests.

Antony’s latest suggestion is typically thought-provoking. Our QA expert on one product currently helps develop test cases before we build each feature, then reviews all built tests after they are done. Instead, what if she paired with each developer as he or she builds the tests – developing and reviewing the tests in place, as it were? This is a bit like removing traffic signage in European cities – it should force the drivers (developers) to be extra attentive, instead of just relying on the road markings (test reviews) to tell them what to do. Hmmm…what do others think of this? Please comment!

PairWithUs!

Just finished watching my first pairwithus video. Antony Marcano and Andy Palmer record their pairing sessions as they work on a real project.

First, I have to say this is a really great idea. Many developers will never have seen a real pairing session with Test-Driven Development, autorunning tests, and refactoring tools in use. It’s nice to see someone doing it imperfectly with the usual frustrations and mistakes along the way – I don’t feel so bad about screwing up when I do it myself.

I found a few things in the video confusing at first, especially the background of the project and the tools they’re using. Much of this is explained in the project page and one of Antony’s blog posts but a brief reference to these at the start of each video would help.

I also have to wonder why they’re not using baby steps – in the session I watched, the test bar was red for nearly twenty minutes! Maybe this was an anomaly – they were working on a particularly tricky piece of code that had to ask the operating system for process status (but why not mock that bit?)

Don’t let my quibbles put you off the videos though – they are really great. Next time someone asks me what TDD is, http://pairwith.us will feature prominently in the answer.

The holy Grail of logging?

So, it’s been awhile since I last posted. Besides some personal time, one of the big reasons that I have been posting less is that we have been heads-down in releasing a new set of features. Many of them involve a set of services built using the Grails framework.

An editorial about Grails is probably due at some point once we get past the honeymoon period. (Right now, things are great!) But, I thought that we needed to post about our issues with Logging and Grails, and how a little upgrade makes all the difference.

In Grails 1.0.x, we struggled mightily to get logging to behave for us. We deploy our app as a WAR file inside of Tomcat, and simply want most of our logging to go to a named log file in the standard Tomcat log directory. All of our environments generally had their Tomcat installed in a different directory. This is especially true of our “test” environments, as we are heavy CI users with many flavors of testing.

Our Grails 1.0.x setup was kludgy to say the least. I won’t even post sample setup files, but suffice it to say, we needed to map all of our logging to two different files. One file was available using grails commands like grails test-app or grails run-app, which could pick up local environment variables (like CATALINA_HOME). The other log file tended to be available when we used a pre-constructed WAR file, which required a fixed Linux path available on most of our production and production-testing environments.

All of this Grails 1.0.x kludginess centered around the fact that creating WAR files compiled your Config.groovy into a Log4J properties file, rather than using the original Groovy class. Consequently, it was very difficult to use that WAR file on many different machines (with potentially different file system setups.)

Now, along came Grails 1.1, which among many other things included a new Log4J DSL to the Config.groovy. So when using the new Log4J DSL, you actually embed your configuration as a closure that the run-time app evaluates at startup. Consequently, you can use Groovy snippets to gather runtime properties to define your Log4J setup.

This made our configuration quick and easy:



log4j = {
     appenders {
         rollingFile name:"mainLog", file: System.getenv("CATALINA_HOME") +"/logs/appName.log", layout:pattern(conversionPattern: '%d %-5p [%t] %c{2} - %m%n')
     }
     root {
         warn 'mainLog'
         additivity = true
     }

     error  'org.codehaus.groovy',  // all
            'org.springframework',
            'org.hibernate',
            mainLog:"StackTrace"  

     warn   'org.mortbay.log'
     info   'grails.app'
     info   'grails.app.controller'
}

Code Dojo: Java Message Service

Our most recent dojo had a dual purpose: first, for newcomers to youDevise (such as yours truly), to get a feel for the types of test refactoring more established developers like to see; second, to introduce developers to the basics of the Java Message Service.

The Java Message Service is a (relatively) platform- and provider-independent messaging standard for asynchronous and reliable communication between Java programs, whether they are in the same VM, on the same machine, or widely dispersed. It supports both the queue (or point-to-point) and publish-subscribe message models.

Because we had a smaller number of developers participating, we ran this dojo more informally than previous ones, though still as a sort of Randori variant. We went through pre-prepared unit tests for classes using the JMS API and passed the keyboard around the table, refactoring as we reviewed the different uses of the API. We used MockRunner to create the unit tests.

After everyone had had the chance to suggest and implement a refactoring on the test cases, we went through a simple use case using an actual JMS provider (Apache ActiveMQ).