Continuous Integration – emphasis on tests not builds?

Continuous Integration – emphasis on tests not builds

(image taken from Jurvetson’s photostream)

Eric Minick just wrote this post on how Continuous Integration is fixated on the act of building, and not testing. It’s interesting: he posits that we all talk about builds, and wrap our tests in builds, kick off builds to do deployments to QA environments, etc. when what really gives us value is testing.

Eric works for Urbancode, who develop AnthillPro. There, they take a different approach to CI, and it’s compelling: with AnthillPro you can deploy and/or test the same build many different times. In the post he’s pointing out that in most CI implementations, the software gets compiled over and over again as you do further testing:

How does one relate the results of various slow builds to each other and the results of the commit build? These types of problems tend to be difficult to solve, often requiring a good deal of cleverness, the use of excessive source control labelingtagging, and at best remain only partly solved. Further, in order to practice CI speed is of the essence; however, the staged build approach reruns the same compilation several times when running different tests with each build. The extra building wastes resources that could be running tests.

I spent a reasonable amount of my time at ThoughtWorks frustrated that we didn’t really solve this problem. We wrote papers about the Build Assembly Line but didn’t manage to translate that thinking into implementing a multi-stage build that we could use on projects to use the technique for allowing the use of exhaustive automated functional tests, which TW was a penchant for.

Compilation, which Eric mentions in the snippet above is less of a problem for me. It’s not too hard to keep the build artifacts around – in fact if you do venture down the path of multi-stage builds then you really want to know that everything compiles in the first step (it’s really annoying to see the second stage of the build fail with a compilation error). Even if you don’t buy that, compilation isn’t generally the thing that cripples projects – it’s slow tests.

He’s pointed to the Martin Fowler article on CI as the source of this influence in thinking. I’ll be interested to see what response ThoughtWorks have, especially as they will shortly be a competitor to Urbancode.

Anyway, well worth a read.

Link

3 thoughts on “Continuous Integration – emphasis on tests not builds?

  1. Douglas Squirrel says:

    At youDevise, we have a two-stage pipeline. The main build compiles everything, as you suggest, as well as doing static analysis and unit tests. Then four other machines take the result, deploy it, and run functional tests. (Actually, I should say machine clusters, since we have several slaves parallelising the work at each stage.)We implemented this about a year ago when we were trying to speed up the pipeline and it seems to work fine; average checkin-to-result time for the second-stage tests that matter is under an hour (though some less-critical tests do take longer).

  2. Julian says:

    Douglas, thanks for sharing. Are you able to say just how the pipeline was implemented? For example, what actually triggers the functional builds and can you easily reconcile them with the fast builds?

  3. Douglas Squirrel says:

    When a product passes its main build, we put a complete build artifact (a .war file, ready to deploy) in a shared folder. When a functional build server is ready for new work, it looks in the shared folder for a new artifact to test. If one is ready, the functional server copies the artifact from the folder to a local directory, then installs it and begins tests. We wrote a little CruiseControl plugin that reads the master log file and identifies the modifications that triggered the main build, so these show up in CC just as normal and you can work out whom to blame – I mean, who should look at a failure. If the functional build is very slow and the product built multiple times on the master (some of our functional builds are on rather slow servers), the plugin is smart enough to identify all the intermediate changes.If we ever really needed to, we could link up a functional build with the triggering master build by looking in the log file (the ant task says which file it is copying, and that file carries a timestamp that identifies the main build). In practise, the list of modifications are sufficient to do the trick.

Comments are closed.

%d bloggers like this: