Monthly Archives: June 2009

CruiseControl.rb gets official Git support

I’m not a fan, but this is about time:

We are happy to announce the release of CruiseControl.rb 1.4.0. This release adds support for three distributed version control systems – Git, Mercurial and Bazaar – in addition to Subversion.
CC.rb remains easy to install, pleasant to use and simple to hack. Since the source has now moved to a git repository, it is easier than ever to fork and contribute. We’re looking forward to your pull requests!



FishEye and Crucible get social

It’s 23:30 at Build Doctor HQ, so this is a quick post: Atlassian announced a couple of new releases today. FishEye 2 and Crucible 2. Both come with social features, which is interesting. From the press releases:

Since code review is typically an iterative process — you code, someone else comments, you code again, more comments.. etc. — Crucible 2 allows you to easily incorporate code revisions into the review process. Simple navigation let’s you step through code changes and/or comments, tracking what has been read and reviewed, letting you easily compare any number of revisions along the way. Crucible’s goal is to make the code review process as easy as email; ideally, something you do for about 10-15 minutes everyday.

… and …

FishEye 2 has a completely new user interface that brings GUI efficiency to web based source code browsing. Every source artifact has a unique URL, everything is queryable, and there are cool charts and reports. FishEye 2 is now faster to navigate with improved usability to make it easier to understand your context. It also has better integration with Atlassian tools. For example, FishEye pulls in activity from related JIRA issues and Crucible code reviews and lets you organize projects across multiple repositories.

Atlassian have really been busting ass to release loads of new developer tools recently. They might do a demo for me, which I’ll post.

Right now the developer tool that we need at the day job is an espresso equipped Piaggio Ape. I’d like to see one on our fourth floor office.


Dtruss: Strace for OSX

(I just found this in my drafts, from 2009)

If you’re an old-school Solaris admin like me, you’re be used to running the truss command to see what applications are doing under the hood. Truss shows you what system calls a process is making. This is incredibly useful when you’re trying to diagnose library loading issues or timeouts, because you can see what the errant process is attempting to do.

Linux has strace, and Mac OS X inherited ktrace from the BSD family. They were all different, but incredibly useful to the troubleshooter.

Starting with Solaris 10, Sun upset the applecart by introducing dtrace. Dtrace isn’t a tool, but a complete language. Apple furthered this by nicking it (well, it is open source) and putting it in Mac OS X, while yanking ktruss.

Anyway, if you find yourself on a system that doesn’t have ktrace, don’t panic. This guy wrote an implementation of truss in Dtrace, and it’s part of Darwin:

knox:~ jsimpson$ sudo dtruss ant
SYSCALL(args) = return
madvise(0x7FFF8AD8E000, 0x2000, 0x5) = 0 0
open("/dev/dtracehelper", 0x2, 0x7FFF5FC45348) = 3 0
ioctl(0x3, 0x80086804, 0x7FFF5FBFD460) = 0 0
close(0x3) = 0 0
stat64("/usr/lib/libstdc++.6.dylib", 0x7FFF5FBFC870, 0x7FFF5FBFCEB0) = 0 0
read(0x3, "Darwin\n", 0x80) = 7 0
read(0x3, "", 0x80) = 0 0
wait4(0xFFFFFFFF, 0x7FFF5FBFE32C, 0x1) = 90163 0

All good.

Atlassian Giveaway Winners!

Thanks to everyone who wrote a story for the Atlassian Giveaway. The entries are in, the judging is complete, and here’s the results!

First prize goes to John Martin, who wrote a story of despair and hope around Continuous Integration. John’s an Atlassian user already, with two other products, so he’s happy to make the hat trick with a Bamboo license.

I wish I had two more licenses to give away to Wolfplusplus and Daniel Spiewak. They both had great stories. Martin Scholl had an amazing tale of folly from a telco firm who should have known better.

I wish I had a special prize for Patrick Debois, who came up with this inspired entry.

Tshirts go to the guys above and: Graeme Gillies, Chris Melikian, Andy Yates, Neil Tingley, and two readers who had to remain anonymous. I’ll be asking for your addresses to hand on for fulfilment. Huge thanks to the Atlassian crew for sponsoring this giveaway. I know they’d be grateful if you took a few minutes to see what they are doing with Bamboo and the rest of the developer tools that they sell.

Here’s John’s winning entry:

I think my best experience with continuous integration had nothing to do with integration _testing_. When I joined the project as the cm/build guy, they were in a cycle of work real hard for two weeks then give the stuff to the build guy to build on Thursday evening. It had to be Thursday because all day Friday and sometimes the weekend would be spent trying to get the thing to actually compile. Typical stuff: developers checked in ‘simple’ changes without compiling and testing, they didn’t always update to get the others’ stuff, they forgot to add new files to source control, they worked in windows and our server was linux, maven dependencies hadn’t been added to our local site, etc. Tracking these down and running around from developer to developer was time consuming. And then the deployment to test could take another half day because something would be wrong with the way the EAR was generated and it wouldn’t load into test’s weblogic server. And, of course, to be honest it wasn’t always the developers’ fault — sometimes we didn’t make a necessary change to the server or upgrade java or whatever.

We introduced cruisecontrol simply to run a compile on every checkin, and within a month, official test builds were smoother because compile problems appeared immediately upon check in _and_ we knew who to talk to without all the hassle of layering everybody’s changes in at 5pm on Thursday. By the time we did the official build, we hardly ever saw a compile problem again. Then we added automatic weblogic deployment (locally), just to prove the EAR loaded, and we could build and deliver to test almost immediately upon code freeze.

That particular project never got to the point where it was using the continuous build for any real test coverage. Two years later, the new build guy asked me why they should still bother doing the cruisecontrol thing if it wasn’t testing. He hadn’t seen an error in a year. I told him that testing should really be added, but even so, if he took away his continuous build, he was going to have to learn a little more about reading stack traces.

Tagged , ,

A solution to broken Gems?

RubyGems have been exceptionally successful as a way for Ruby developers to share code. We generally think that sharing code is a good thing. Certainly, the Rails community can write projects exceptionally quickly; thanks, in part to RubyGems.

My beef has been that RubyGems doesn’t play nice with operating systems. We have standards for where things should live in any operating system. In the world I inhabit most (Unix systems), there’s a well defined place for everything. This is more than just being tidy. It helps keep things consistent and stable: when you install new software, dependencies can be easily found, for example.

Gems don’t really work like that. You can have many gem installations on your system, and they can all provide code to an application. It’s for this reason that the Debian team took the RubyGems code, modified it, and put it in a Debian package. This helped bridge the worlds of Debian packages (a packaging system that does dependency resolution to work out which packages you need in order to run the packages you want) and RubyGems (which mostly does the same thing).

Unfortunately this often lags behind the rapid development of RubyGems, so we’re forced to do things like install the newer version by hand over the top of the Debian version. This gets the job done, in doing so breaks the packaging system, which is a key reason that people use Debian or Ubuntu. Think it’s just me complaining about this? Some other people who know a lot about operating systems aren’t happy either:

At least for Debian or Ubuntu users, there’s one solution: DebGem. DebGem (from the guys who brought you Phusion Passenger and Ruby Enterprise Edition) bundles RubyGems inside Debian packages. If each package has its dependencies declared right, you’ll be able to install any gem and it’s native dependencies, in one go.

I hope it works.

The issue here is that it takes lots of time to maintain packages. Especially when you’re trying to track the packages that come from an extremely prolific community that knows how to use distributed version control systems (git is now the VCS du jour for Rubyists). Let’s hope that Phusion can charge enough to make it worth their while to continue; once you’re hooked on this, it would be hard to stop.

Not sure that this really solves the problem. The core issue here is that the developers are elegantly solving the problem of how to reuse code as developers. Systems administrators are trying to make systems that are stable and easy to maintain. The two might never meet.

Final thoughts: they might do well pursuing a freelance model to get gems packaged. Also, a commercial distribution might buy it to gain market share. My fee for this idea is a snip at 10%.

( image from Ed Yourdon )

Deadline for Atlassian ‘build horror’ entries is Friday.

Get them in before midday, BST. The next thing you hear on this matter will be the announcement of the winners.


Build Pattern: The Captive Build Tool

Check your build tool into your version control system. Ideally you’d do this in a relative location to your project(s). That way you can have a go.bat or a one line wrapper script to call the correct build tool from your project. Don’t get clever. This should be the simplest script you can manage.

Once you have this set up, a new developer can be cutting (and building) code on their first day. This is a huge boost for your newbie developers. This pattern brings more love to your team because a new library doesn’t mean all of your guys downing tools to add it.

I was going to call this pattern Ant Farm, but that would have been a little Java specific. We’d also need NAnt farm and Phing Pharm. So why do this? The build tool should be vanilla enough to deal with any project. If that’s the case then it becomes a commodity on the developer’s computers. However in this messy world it never seems to work that way. Here’s why:

  • Someone will have to use a new feature, which calls for everyone to upgrade the build tool
  • You’ll end up using a task which insists on being on the boot classpath
  • One day you’ll look at the build tool and find half a dozen key dependencies are residing in it.

So go with the flow. Make one canonical build tool and make it a damn good one. Put all the useful tasks in it. Yes, all of them. But with one exception. Make sure that they aren’t project specific. Do you want to have to have the right version of build tool to build and test your application? No. Didn’t think so.

Handy Elements of Enterprise CI wall chart

Fresh from the UrbanCode guys. Our copy is on the wall. There’s even an A4 version for us Europeans. I have to admire a company that does the metric system.


(photo by Emma Louise)

What do you mean, ‘integrate’?

Here’s my entry for the Atlassian Giveaway. I’m not eligible to win a prize, but I’m sharing the tale anyway.

Sometime in the 90’s a former employer of mine had an Intranet. The intranet grew from a simple repository of pages, to something with much more bling: there was an address book, a search engine, and some simple tools for trading, written in HTML and liberally mixed in scripting languages.

A few years later, the trading tools were coping with millions of pounds’ worth of business. There was concern that the present version control system wasn’t adequate for the task, so they used a popular enterprise tool. One of the developers cheerfully took everything that was in the web root of the Intranet server, and checked it into the new VCS. Job done.

So there was now a project that was approximately one gigabyte in size. To change content, the business would make a request, a developer would make the change, and place the related files onto a shared filesystem. He or she would raise a change request to go past the change control board, and if it were approved, the operations team would drop the files into the production webroot. So where’s the horror?

Because the project was so big, the developers wouldn’t use branches to track what was in development and what was in production. So they made the changes locally, and kept them on their PC’s. When a change went into production they would check the files back in. So integration happened *after* a change went live. Which isn’t the normal order of things.

The other terrifying thing was the sheer number of URLs inside the code that had to be edited at deploy time. Deploying to ‘’ meant replacing every url inside the code, by hand.

Why was the project so big? It was composed of:

  • dynamically generated print output,
  • digital photos from the last 7 years
  • documentation from long-dead projects
  • general cruft

About 10% of the project was useful code. I got approval to fix a lot of this when the business tested a feature, clicked a link, and hence made test trades in a production system. Ouch.

Think you can share a tale and win a T-shirt? Leave or a comment or send an email to Hurry though, we’re closing entries soon!

Tagged , ,

One scary deployment – Bamboo giveaway preview

We’ve had some entries in, and I thought I’d share the one that horrified me:

My last gig I was hired on to be the build guy. On my first day I sat
shotgun to their deployment process. The manual process was as
1. Logon to the ‘build box’
2. Get latest
3. Open visual studio and compile the application
4. FTP the resulting app to a staging area on our production webserver
5. Put the website offline
6. Run any new SQL files against the production database (Hopefully
you guessed the execution order correctly)
7. Copy the app into place
8. Put website back online
9. Hope nothing had broken.

Staggering. Do you think you can better it? Send us yours!