Monthly Archives: November 2008

Effective Functional Web Tests

Douglas Squirrel just posted about the issues he’s having with slow functional tests. My last big project was a large website that used Selenium for functional testing. I thought I’d share what worked:

  • Make damn sure that Selenium doesn’t spawn a new browser. If you use (for example) JUnit tests that call Selenium objects for all your functional tests, make sure you don’t spawn a new VM with your tests. That will certainly cause Selenium RC to spawn a new browser. Fixing this dropped the test execution from 30 minutes to 15.
  • Don’t let the pages render any external objects. Generally you’re just trying to prove that your pages render with enough form that you can trace some functionality through them. Displaying ads or any other external component will just get in the way. If you can’t cause your pages to render in such a way, you can use a firewall to block those elements. You can’t just drop the packets, however. You need to kill the entire TCP session. The fantastic infrastructure architect at this project used the Iptables REJECT commmand to make the browser give up immediately on rendering the external components. This sped up the tests a little and more importantly made them rock solid.
  • Make the first failure fail the build. When the functional test run was 30 minutes, you could be left waiting up to an hour to find out if your checkin had caused a test failure. The ah-ha moment came when we realised that we could fail the build immediately the moment a functional test failed. That gave feedback to the rest of the team that the build was broken. We learned to be very disciplined about checking in our code.

CI Cage Fight – Cruise

Note: I’m not going to even pretend to be unbiased about this one. I used to work for these people. I was involved in hiring Chris, and I was involved in some early discussions about previous incarnations of this product.

Chris Read presented Cruise, from ThoughtWorks Studios. Loads of people think that this product is the payware version of CruiseControl. If Cruise is a relative of CruiseControl, they are third cousins, once removed. And they don’t look alike. It’s almost completely a from scratch rewrite, and by the time I write this, the last bit of CruiseControl code should be gone from it.

Cruise has all the features that you’d come to expect in a commercial CI server these days: distributed build facility, whizzy UI, repository of built artifacts, etc. There’s two features that I’d like to talk about: the Build Pipeline and the movies.

When your CI build starts to get uncomfortably long, what do you do? Assuming you can’t shorten it, you can break it up into stages: one for developers so they know that their unit tests passed, another to demonstrate that functional tests pass. If you want to get really fancy, you can map to each stage of the QA process. This whole idea has been dubbed “The Build Pipeline” by ThoughtWorks and it seems like a reasonable implementation.

A couple of the key aspects: you can retreive built artifacts from the first build to use in later builds. So no need to rebuild artifacts or build your own repository. Also, the tool tracks all the builds so you can see that one developer checkin spawned all these builds. That’s mighty useful.

The other useful feature is that it records Flash movies of your Selenium tests. That may not sound like much, but have you ever tried to diagnose a broken Selenium build when it’s not on your PC? I have sat there eyeballing the thing waiting for it to fail. It’s not the kind of feature that you CIO will froth over, but your developers will love it.

Cruise costs money and is closed source software. Two build agents are free. There’s a bunch of other features. LDAP integration is always a good ‘un.

London Build and Deploy Job RSS Feed

Another day at the office

Short story: I made an RSS feed for Build and Deploy jobs in London. I’m going to run it as a public service until January and see if it’s helped anyone. If I help one person find a job in that time, I’ll be very happy.

Long story: I get asked if I know anybody who can do build and deployment work around London. Generally I don’t. I’m happy to blog about them, but as my readership is mostly in North America (rankings for last month were 1. USA, 2. UK, 3. Germany and 4. Canada) I wanted to come up with a way to help, but keep the content relevant to the majority.

Microformats were the first idea that came to mind. Keep a page with job listings in a microformat, scrape the page and render into RSS. I started reading up on it but decided I’d park that. What worked, and what I could write offline on the train on my eeepc was simpler: a YAML file of jobs, and some Ruby scripts to generate an RSS feed. CruiseControl pushes the XML file of RSS feed out when there’s an update, and Feedburner does the work in counting subscribers and allowing me freedom to change the backend implementation. Beats reading Metro.

Anyway, that didn’t scale, so I now use a commercial job board tool and a Yahoo Pipe to filter out just the London jobs. Yahoo Pipes are great.


(image taken from foundphotoslj’s photostream)

Too Agile

Easy to say the ‘A’ word. Harder to do.

The Build Rosetta Stone

Update: I had some nice feedback from @jtf @nasrat and @cread, so I have pushed the source files to a git repository.

What’s the equivalent command to [some command in your usual build tool] in [some build tool that you’re using now]? I found myself asking that question too often earlier this year, so I set out to make a tool that would help. How does it work? The tables of commands are stored in a Markdown file, which is rendered into HTML and PDF with the sweet Maruku ruby library. It’s built via CruiseControl and published to the Apache docroot if it generates successfully.


(Image taken from bortescristian’s photostream)

Tagged , , ,

CI Cage Fight – Hudson

The next installment in this series is Eric Lefevre presenting Hudson. Hudson is an open source CI server that has gained a fantastic amount of ground in the past year or two. Out of the box it has a pretty good feature set:

  • Distributed builds. You can declare a dependency on a particular OS and Hudson will route the build to an agent that satisfies that dependency
  • Very good interface (that damned butler!) with a nifty weather metaphor for build status.
  • Localised in 8 languages
  • GUI based configuration
  • You can declare an entire matrix of different builds and let Hudson work out the details. (Fitnesse build on Windows, Fitnesse build on Unix, coming right away sir)
  • Plugin-friendly architecture and a pretty good community of plugin developers – very smart move

There’s a load of other features at the Hudson website. What I didn’t note down is that it’s amazingly simple to get started with. I hope it raises the bar for CI servers, commercial or not.

Tagged ,

CI Cage Fight – CruiseControl

The second presentation at the CI Cage Fight was Paul Julius, talking about CruiseControl. I think of CruiseControl as the daddy (or at least an uncle) of Continuous Integration servers. It’s the oldest Java CI server that I know of. As an open source project I think it opened the way for projects to adopt automated CI.

One of the unique features is the support for so many version control systems. I believe it supports more VCS systems (everything from Accurev to VSS) than any other CI server. CruiseControl is also very seasoned: it’s been deployed in many organisations (over 130,000 downloads for the latest version) and a lot of those inevitable wrinkles have been ironed out.

The two other things that I noted down from Paul’s talk were the flexibility (remember, CruiseControl was written as a framework, not a product for sale) and the useful file based configuration. The parsing of the XML file is very good. I haven’t had many situations where I have managed to confuse the server so much that it couldn’t tell me where the errors in the file were (I can’t say that about any other CI server). It also has some brilliant features for reducing duplication in the file, which I have gratefully used.

Another interesting fact is that Paul and JTF‘s collaboration on CruiseControl probably gave us CITCON, but that’s another story.

Tagged ,

CI Cage Fight – build-o-matic

One of the sessions for Citcon Europe 2008 was Tom Sulston’s Continuous Integration Cage Fight. Seven geeks lined up to show off their CI prowess, and the first one in the cage was Ivan Moore with his CI server, build-o-matic.

Build-o-matic is an open source CI server written in Python. It’s classic open source in the sense that it’s there to scratch an itch: none of the other commercial or open source CI tools will really get to the bottom of who broke the build.

When several developers check in at once and cause a train wreck of a build, it can be hard to see which individual checkin made the build fail. Build-o-matic will do a binary search over the revisions that make up the build, then build them individually until it can find the offending checkin.

Another nice feature is that you can rewrite history to a certain extent and re-run a build. Someone kicked out the Ethernet cable to the build server? No problem. Re-run the build.

My favourite feature though, is the faces. Build-o-matic can display mugshots of the offending developers. I’m not aware of any other CI server (comments please if you do know) that does this. Ivan says it does have a positive effect on team discipline, and broken builds got taken more seriously. It puts me in mind of this paper.