Monthly Archives: March 2008

CruiseControl Enterprise Best Practices, Part 6: Scaling up

You just started out with Continuous Integration. You’re building your project in CruiseControl. Great. Now, it’s time you started to plan ahead. This post is about scaling up CruiseControl. The tool can scale up to many projects, but you have to know a few things.

The very first thing to do is make sure that you can simultaneously build your projects. CruiseControl uses Java threads to manage builds and projects. Each project has it’s own thread. There is a least one Builder thread that will actually run a build, and there is a Build Queue, which mediates the requests from the project threads to the builder thread. You need to give CruiseControl enough builder threads to work with: by default it gives you one thread. The impact of this is that if you have 5 projects configured, you’ll only ever build one at a time. This could be good if you have a tiny build server, or projects that don’t play nicely together.

On the other hand, if you have plenty of capacity to use, you will get a boost from the extra builder threads. The right setting for the number of threads will change over time; don’t hesitate to do some performance management and find the right value for your server.

      <threads count="2" />
<!-- rest of config file suppressed -->

Jeffrey Frederick wisely points out that it is useless to set the number of builder threads to be higher than the number of projects that you have.
There are plenty of other factors to consider when scaling up your Continuous Integration service: the speed of your Version Control System, where you store artifacts, logs, etc. One factor that I will mention today is disk bandwidth. Building software is a disk-intensive process. Even when a build is done compiling Java code and making archives like jars, CruiseControl has to write logfiles. It’s very easy to overload a single disk with all this activity. Ideally you want to make sure that your machine can build several projects at once.

On one of my very first projects at ThoughtWorks, the team was running two CruiseControl servers, apparently because the first Solaris server was too slow. That didn’t seem right to me, so I dug deeper. With my systems administration background I was able to see that a single disk was 100% busy running the operating system, CruiseControl and the build. I spread the workload across the four disks in the system and the machine was able to manage many more projects.

This pattern has repeated itself on many of my subsequent projects. In my experience not many CI servers are constrained by processor overhead. Unfortunately, it’s often painful to rectify disk issues once a system is up and running. Make yourself some luck and order plenty of fast disk drives before you scale up and people start to complain about the slow build.

The way to implement this is to use the configuration file. The log element is used to tell CruiseControl where to write logfiles. By default it’s a suubdirectory of the CruiseControl installation. But if you change the value of the dir attribute of the logfile, you can make sure that the logs are being written to a disk that isn’t already running CruiseControl. If you’re using Ant to build your code, you can use the antWorkingDir attribute on the ant element in the config file to make sure that your projects are built on another disk.

I can’t really do a good example for this one as each CruiseControl instance is so different. Buildix is an attempt to make many installs more homogenous. If you look at the way things are laid out on the disk, you’ll see that the CruiseControl install is in /usr/share/cruisecontrol, but the projects and logs are installed in /var/spool/cruisecontrol: the reason we did this was so that you could mount the /var directory on another disk if things got busy. Drop me a comment if you want to know more.


New, shiny Rake support in Team City

You couldn’t pay me to run CruiseControl.rb now. Oh wait. You probably could.

Anyway, now that Team City has a ‘Rake Runner’, you can build your Ruby projects with Rake. It understands Test::Unit and Rspec test output as well. The runner gives you test timings, both for that build and a graph of timings in recent builds. It also gives you Rcov support as well. You can have a look on their public demo server.

It’s available as an EAP.


New colourized Logger for Nant

Update: It was of course Eric Liu who wrote the logger. Stephen Chu wrote this adjacent post about Rails. Sorry, Eric and Stephen. Thanks to the people who left comments pointing this out.

I spent several years being sysadmin at one project. Possibly the most popular thing I ever did there was enable the Ant AnsiColorLogger for all the developers, giving them highlighting in the IDE and the build.

Anyway, Stephen Chu from ThoughtWorks just wrote the same for Nant. Nice work.



Ant Best Practices: Provide good help

This is the fourth post in my series of posts examining Eric M Burke’s
Top 15 Ant Best Practices from December 2003. The last one I covered was ‘Prefer a single buildfile‘. Today’s is ‘Provide good help’. Eric starts by suggesting that you should make use of public targets to document the targets that you should be able to invoke.

Now, the distinction between a public and a private target in [N]Ant is pretty subtle: If the target has a descrption, it’s public and will show up when you call Ant with the -p or the -projecthelp option. If a target has no description, it’s private. So, the description attribute signifies that the target is for public consumption; and also has the side effect of telling you what you can do with it. Using this mechanism is a no-brainer – but how do you protect a private, or internal target? There’s no way that you can declare a target as private, but you can stop the target from being called from the command line by prefixing it with a hyphen.

For example, if you have a target called start-weblogic that you don’t want called from the command line, you can call it -start-weblogic. If you try and invoke it, look what happens:

graham-tackleys-mac-mini:~ jsimpson$ ant -start-weblogic
Unknown argument: -start-weblogic
ant [options] [target [target2 [target3] …]]
-help, -h print this message
-projecthelp, -p print project help information
-version print the version information and exit

Eric also suggests 2 other things: using XML comments to provide more detail, and making a help target that echoes out some useful help messages. I don’t think I’d ever disagree with a help target. I might disagree about the XML comments. Generally I use XML comments as temporary notes to myself. It’s nicer to use the description attribute on elements in build.xml files: just about every element will accept one, and you can make sure that the text always matches the element.

<project name=”MyGreatProject” default=”test”>
<property environment=”env” />
<property file=”” description=”this gets all the special properties”/>

So far, that’s about 4 out of 4 practices that have held up. Next: clean targets.


My article on Refactoring Ant Builds is published

The Pragmatic Programmers have just published the first volume of the ThoughtWorks Anthology! My article on Refactoring in Ant is included: I distilled my experiences fixing broken Ant builds for ThoughtWorks into a collection of refactorings that can be applied to a crufty build. You can buy PDF or paper copies via the Prags, but if you want to buy it via Amazon, there’s a link on Build Doctor.

I’ll write more about the process of writing an article on my personal blog.

Link (via Josh Graham)

Tagged ,

renaming multiple files in Perforce with the help of Ruby

Half an hour ago, I had about 60 files that had the same irritating prefix to the filename. I wanted to drop the prefix so the reader could see the significant portion of the filename first. There is no way that I would rename that many files by hand and not mess one up.


def rename(file,path)
new_file = file.sub(path,”)
system(“p4 integrate #{file} #{new_file}”)
system(“p4 delete #{file}”)

Dir[“*.build”].each do |old_file|
rename(old_file,bad_path) if old_file.match(bad_path)

Two minutes spent playing with ruby goes a long way.


Ant Best Practices: Prefer a single buildfile

This is the third post in my series about Eric M Burke’s Top 15 Ant Best Practices from December 2003. Today’s practice is “Prefer a single buildfile”. To paraphrase Eric, it’s easier to understand a single file rather than a clever hierarchy of files. Having presided over several multi-file builds, I’m only too aware that you can easily hang yourself. On the other hand, sometimes it makes sense to extract something to a different file to achieve a separation of concerns.

If you are trying to map complex dependencies across many build files, or do amazing things in Ant or Nant, then perhaps you need to try and simplify things in your project. I once worked on a project where the frontend was written by my team, and the backend by other. The frontend team branched the work of the backend team, and the backend team kept on trucking with their work. I came to the belated conclusion that we had fallen prey to Conway’s Law; really we had one project, and we should have written it like that. The build on that project was a nightmare: two projects pretending they were free agents, but in reality hopelessly dependent on each other.

I’m torn on this practice, but I’m going to agree with Eric. Ideally you shouldn’t need multiple files. Though it does keep me in a job.


Ant Best Practices: Adopt consistent style

In my first post reviewing Eric Burke’s Ant Best Practices article from 2003, we looked at putting the build file at the root of the project tree. I had a bee in my bonnet about that one; and so I skipped Eric’s first point, which is to adopt a consistent style.

The man had a point, and it’s still true. Some build files would appear to be an amalgam of different styles. Maybe I’ve been doing this too long, but I could tell who wrote certain parts of the build at my last project without looking at the version control system to find out. Just like Java code, you need to decide on tabs or spaces to indent with, and how many tabstops/spaces to set. If you’re using Eclipse, you can configure this, and check the .project file back into your project’s source repository so any Eclipse user can automatically use the same settings:

What’s changed in the intervening few years between the publication of Eric’s post and this one is just how much better IDE support for Ant has gotten. As long as you don’t do anything odd (there’s several posts in that one alone) to your build, you should be able to navigate through your build using Eclipse or IDEA. Control-clicking on a target reference will take you to that target’s definition, and you can get an outline view, amongst others. You can also run and debug the build from the IDE, which is awfully convenient.

For me, it’s a good reason not to put enormous XML comments in the build, like this:

<!– =================================
target: name
================================= –>
<target name=”name” depends=”depends” description=”–> description”>

They would be helpful if you’re editing your build.xml using a monochrome terminal and a text editor like vi (which I have done). But as IDE’s and editors get better, there’s less call for such things. I put my energies into making things consistent and easy to read. Eric: that’s 2.


Graphing your CruiseControl build times with Gruff

There had been some concern on my last project about build times. My erstwhile colleague Ben started graphing the builds on a sheet of paper and posting it in the kitchen so everybody in the team could see it. When the sheet fell of the wall and the cleaners threw it away, I knew it was time to do something.

Cruise Publisher
How do you get the build time in the first place? I wrote a CruiseControl publisher that called some Ruby code to extract the build duration from the CruiseControl log, which is an XML file. It was a very simple script that matched duration of the Ant build in the XML file, converted the text to a number of seconds, and then made an HTTP POST to a Rails application. The build duration got passed as a POST argument.

We had a Rails application to display other project information. So for us, this was a convenient way to store the information, with the date coming from the database. You could use a text file to store this information instead. Once everything was in place it was easy to write some more Ruby to make the graph appear.

The Code

require ‘chart’
require ‘gruff’

data = { “count” => [ 3, 5, 2 ] }
g = 640
g.title = “Usages of the word ‘foo'”
g.labels = { 0 => ‘Monday’, 1 => ‘Tuesday’, 2 => ‘Wednesday’}
data.each_pair {|k, v|, v)}
blob = g.to_blob

out =“blob.png”,”w+”)
out < blob

This code uses Gruff to actually generate the chart, and you’ll need to install this separately. Pass it some data and it will build a PNG file with your graph. By plumbing it into the radiator, the trend in build times is visible to the entire room. It really helped to keep the team focussed on the length of the build – especially for browser based tests that can easy start to slow things down.


Antcall is evil

Antcall is evil. Trust me on this one. This Ant task will hurt you. Use Macrodef instead.

A brief introduction to make

Make is the daddy of Ant. It originally came from the Unix world but it has been ported to just about every computer system, as it’s the de facto way to build c code. Make reads makefiles, which tell make how to do your bidding. Inside the makefile, you declare targets, declare their dependencies, and let make do all the work. Here’s a simple example of a makefile:

jsimpson$ cat Makefile
a: b c
echo “A: I depend on b and c “

echo “B: I depend on nothing”

c: d
echo “C: I depend on d”

echo “D: I depend on nothing”

The format of the makefile is simple. A target begins with a dependency line containing a name, a colon and an optional list of dependencies. The dependency line is followed by one or more command lines that are indented with a tab. The command lines do the work. In the example above you can see that target A depends on targets B and C, and that target C depends on target D. When we execute the makefile, make determines the order in which it should execute the targets to do the work. Have a look:

jsimpson$ make
echo “B: I depend on nothing”
B: I depend on nothing
echo “D: I depend on nothing”
D: I depend on nothing
echo “C: I depend on d”
C: I depend on d
echo “A: I depend on b and c “
A: I depend on b and c

The make that is installed on my Mac decided that the appropriate order in which to execute the targets was B, D, C and then A. It could have chosen D, C, B and then A. It doesn’t matter because you tell make the dependency rules and then make satisfies them. This approach has been working for authors of Unix software since 1977 , thanks to Stuart Feldman at Bell Labs, who wrote the original make. Make is still actively used on many projects today, but for the bulk of Java and .NET projects, you’ll be using Ant and NAnt. This is okay. Make isn’t exactly cross-platform, and the two tools are evolved with support for building their platforms.

How does this help me write good Nant or Ant buildfiles?

What you are doing when you write a makefile, or build.xml or is declaring dependencies. What the build tool (Ant) does with those dependencies is create a dependency graph. We can visualize the dependency graph of our simple build:

It becomes very important to specify the dependencies of each target correctly. If you write a build target that doesn’t declare it’s dependencies fully or correctly (say, it actually depends on the output of another target, which is run sometimes) then you end up with a broken or unreliable build. It’s an easy mistake to make, especially as complexity on your project grows. But how does all this relate antcall?

Antcall seems okay at first glance. Each use of this Ant task will execute a named target in the same buildfile as the calling element. Which is really convenient if you explicitly want to invoke a target at a specific point in your buildfile. But what if that target has dependencies? The Antcall documentation hints at the pain:

When a target is invoked by antcall, all of its dependent targets will also be called within the context of any new parameters.

If your antcall (or Nant call) target has no dependencies, then you have done nothing wrong but violate the dependency based style of the tool. If it has dependencies, then they will be relentlessly run every time you call the target.

One pattern I have seen a lot of is using the [ant]call to specify dependencies in order, rather than declaring them as such:

<target name="chico"/>
<target name="zeppo"/>
<target name="harpo"/>
<target name="groucho">
<antcall target="chico"/>
<antcall target="zeppo"/>
<antcall target="harpo"/>

In a twisted way I can see why the idea is attractive: You want to be sure of the order of target execution. To be really sure, you need to specify dependencies on the targets. So in this case you’d declare that harpo depended on zeppo, for example.

<target name="groucho" depends="chico,zeppo,harpo">

ant will attempt to preserve the order in which you declared them. The crucial difference is that should one of those targets depend on something else, it can adjust the order of target execution to accommodate it.

Another aspect that I haven’t covered yet is reuse. If you want to try and invoke a target several times with different parameters, the dependency model doesn’t really cut it. Before Ant 1.6 came out, there really was no choice but to use antcall.

Anyway, my advice for anybody using antcall is to substitute it for macrodef, where you really need reuse.

Thanks to T Ashitani for the online dot implementation.

Tagged ,