Alex Verkhovsky describes lots of test-automation lessons in his post on UI automated testing. We’ve learnt most of these the hard way – for instance, we are just beginning to break our dependency on browser-based tests (which are slooooow) and use integration tests that go in just behind the UI. (Link thanks to the Build Doctor.)
I recently conducted an open session on rotating pairs at XP Day. During the discussion there was mention of Arlo Belshee who conducted an experiment on rotating pairs. In the experiment, Arlo found that rapidly exchanging partners was an extremely quick way to:
- Produce features
- Catch defects fast
There was talk about research that Arlo had conducted on how long it should be before rotation takes place. If you take longer and longer to exchange partners, you tend to hit a flat ‘plateau’ which is analogous to productivity going down. If you reflect and graphically see what happened before you reach this plateau, you get this curve (which is analogous to productivity peaking). Arlo found that the flattening of the curve started at every 4hrs.
According to him, it doesn’t matter whether you exchange partners every 1 day, 2days, etc. you only get about 4hrs productivity without swapping pairs. Arlo found that in his 6 person team, by rotating pairs every 90 minutes, he got a:
- 2-1 improvement in velocity (when compared to a 4 hour swap)
- Decrease in bug rate
- Notable improvement in code debt
In addition, the following points were discussed about rotating pairs:
1) Reduces skills delta and standardises coding styles.
Whilst working on a task, pairs play the roles of a ‘beginner’ and an ‘expert’ respectively. When the swap takes place the roles reverse i.e. the ‘expert’ moves onto the next task and becomes the ‘beginner’ in that task and the ‘beginner’, becomes the ‘expert’ in the current task. The role of the ‘beginner’ or the least qualified implementer means that the person believes he/she doesn’t know as much as their peers and think that he/should should. This beginner’s mind is attentive and opens up to learning noticing ‘everything’ and absorbing information as fast as it can. In this phase, experimenting leads to lessons being learnt quickly.
2) Business knowledge transfer and team ownership.
Whilst working on tasks or stories, pairs will have business knowledge transferred to them. This means that developers will be familiar with all features and business knowledge. Also, rather that an individual, there is a set of pairs who specialise in any field. This leads to team ownership rather than individual ownership. However, it was noted that there is a tendency that developers would irritate or annoy business analysts by asking the same question multiple times. Developers may also lack accountability and responsibility.
3) Problem solving becomes easier.
A pair trying to solve a problem may find themselves going down a dead end and not making any progress. Swapping controls this problem with fresh perspective finding a new and better resolution to the problem.
4) Fewer bugs and good design
A code review constantly takes place in order to get the one of the pair members up to speed with the current functionality. This leads to problems, defects being caught early and the design being evaluated and refactored if necessary.
5) Flexibility improves
Pairs working on a feature from start to completion will find that they can’t enter and leave the working environment as their convenience. However, with rotating pairs this becomes easier as individuals can join the rotation at the next iteration.
In agile teams when one person gets a contagious illness e.g. flu, the chances that all team members contract that illness are significantly increased. The number of sick days goes up and productivity goes down and this leads to down time until each team member recovers. However, in the next iteration of the flu all team members will have built up immunity and there would be no down time or sick days.
7) Arguing over best practises
Although it was noted that this was de-motivating for developers and that pairs may never come to a good agreement or complete a task, this was noted as a good practise because pairs would see the need to work together to find the best solution possible.
8) Loss of conceptual integrity.
Pairs may lose conceptual integrity because, it may be the case where the transfer of knowledge to the new pair member is incomplete and this will filter further as more swapping takes place. Also the least qualified implementer in unable to digest all the knowledge passed on to him/her by partners.
9) Features completion.
While it was noted that pair rotation was a good practise that led to high quality and almost bug free feature, from a business perspective, it takes that much longer to deliver a feature. This could lead to a both internal and external customer frustration and a lack of confidence.
We all agreed that what works for one team does not necessarily work for another. We should experiment and see what works. For example, if a team is still arguing with each other in a ‘canonical’ storming phase, then pair rotation should of a moderate nature. As can be deduced from the above discussion, pair rotation has its advantages and disadvantages, but in order to find out whether it is a blessing or a curse for your company, it was recommended that you try it for a fixed period after which a retrospective will disclose the findings.
We have a farm of 15-20 continuous-integration servers (depending on how you count) whose job it is to tear apart every checkin and tell us all the bugs in it before customers see them. Hudson does a great job managing the builds and Selenium Grid lets us add and remove browser-test machines. Our biggest problem is getting fast enough responses – it’s not unusual to wait 3-4 hours or even longer before you get a full set of results from your checkin, by which time lots of others have also checked in and you’ve forgotten what you were working on.
We’ve identified three related metrics that we would like to tune:
- Wait time is like latency in a network – it’s the time your build spends waiting before it gets to run on a Hudson slave. More precisely, it’s
startBuildTime - firstCheckinTime– note this includes time for predecessor builds to run, if you are using a build pipeline (as we are).
- Build time is just that:
finishBuildTime - startBuildTime. In other words, once your build got assigned a Hudson slave, how long did it take to finish its run and give a result?
- Response time is the total time between your checkin and the email from Hudson giving you the result:
finishBuildTime - firstCheckinTime, or equivalently
waitTime + buildTime.
We have started gathering these three values daily for every job we run in Hudson. As we are concentrating on stability at the moment, the results are pretty flat, but we are just starting to add more Hudson slaves (which should reduce wait time) and Selenium Grid slaves (which should reduce build time for browser tests at least). We’re also using Nagios and Cacti to keep track of utilisation on all our servers. Stay tuned for the results!
Unfortunately measuring these things is quite a pain. I’ve had to write a gnarly Python script that parses Hudson logs and even uses file modification times to figure out the three key times
finishBuildTime. If any readers know how to do this better (fancy Hudson plugin maybe?) I’d love some help in the comments.