Monthly Archives: April 2009

Citcon Paris is filling up fast

Citcon Europe ’09 news:

  • We should be able to announce a venue soon
  • It’s two-thirds full! Eric Lefevre has more:

While some are currently enjoying CITCON Minneapolis, the Paris edition is quietly making progress.
It turns out today that we have passed the 100-registrants mark for CITCON Paris. That’s 2 thirds of the capacity. And there are almost 5 more months to go before the conference! This, I believe, is the strongest CITCON registration drive ever.
So, if you have joined us on previous editions and you are considering coming to Paris in September 18th & 19th, register now, before it is too late. Judging by CITCON Minneapolis and Brisbane, this could be the first year we will actually turn away participants.

You heard the man. Don’t miss out!


Continuous Interviews: Jez Humble, Part II

This is part two of the interview with Jez Humble from ThoughtWorks Studios (part one here). I’m also getting it transcribed; watch this space.

JULIAN: Can you tell me about the XML?

JEZ: Yeah, that’s…

JULIAN: I was looking at Cruise the other day, because I was wanting to change who we are using at my day job, and yeah, I was quite surprised to see that after you get through the wizard of creating a pipeline, it sort of reverts to XML; what’s all that about?

JEZ: Yeah, so that has been a frequent source of complaint. So, there are some advantages to XML config, which is that it allows for simple, external configuration of the tool. So, you could, for example, send the configuration out via puppet and manage it centrally in version control, and it becomes much easier to kind of audit it and see exactly what the configuration was at a particular point in time. Obviously, it is extremely un-user-friendly. We have put some little things in there to help you with the XML config; so, for example, if you use the wizard in Cruise, it does not let you save the config; it is not valid. So, there are some bits that make it slightly nicer, but it is still horrible, obviously, and we are committed to getting rid of it. What we are planning to do, is to put wizard in, over the course of, you know, the next few releases. So, hopefully you will start to see that with 2.0 general release, at least some incremental move, and we will be moving everything over to wizard. What we decided to do, is to focus on delivering other functionality before delivering wizards. I mean, it is what has enabled us…

JULIAN: Understood, fair enough…

JEZ: Yeah, but I know that it is horrible.

JULIAN: Yeah, I mean, you absolutely need some kind of easily causable config file, and being able to push that out with some of the management symptom is brilliant. I mean, I do that on my NCS server, as all the config is managed under subversion, using a puppet. Fair enough; it is good to see that you will be moving along from that in time. I would like to talk about the build pipeline if you can. I mean, obviously, you and I have discussed this over the course of 7 years, but for anyone who has really just come along and has seen the **2:22**, can you kind of give a brief overview of that, and what it really does for you on a project?

JEZ: Sure. So, obviously, every project has this cycle of building, deploying, testing, and releasing, and all your teams would be involved in that. And really, the problem is, as we might have seen, it is very hard to get visibility into that process, and to manage it well. So, it is very typical for developers to throw things over the wall, and then for build dudes to have to come and kind of try and do manual deployments, and that takes ages, and then the developers don’t get feedback on the deployment process for ages, and then when they get bug reports, there are 2 versions behind the development, and the bugs are not relevant anymore, and then the operations people are up until 4 a.m. trying to deploy it. This is very typical, and this is the problem that we are trying to solve with Cruise; actually, to give everybody who is working on a project from operations to program managers to developers visibility into the status of every single check-in, which automated and manual tests it has passed, whether it has passed, performance tests, other kinds of nonfunctional tests, and to get control over that process, so that you can do click-button deploys. You can just press a button, and bang, it is in your manual testing environment; bang, it is in your staging environment, and, ultimately, pushbutton deployments into production. One of the interesting things that has being developing over the last few months is kind of continuous deployment, and I think Cruise is really tool that is absolutely designed with that in mind, the idea that any check-in, you can see the moment that tests are passed, and you can just press a button and have that deployed into production, straight away.

JULIAN: Okay, great. Um, the last question I was going to ask was about one other kind of side-effect I have seen as being able to have all these different stages of build, which is projects ending up with some kind of horrible automatic functional test which might run for a very long time. There was a major retailer project that was done at ThoughtWorks that had a function test suite that ended up running for hours, and I think it was nearly approaching days at one point. Do you have any comment on what you should try and do to avoid that? I mean, it is quite easy to generate yourself an enormous volume of functional tests, any kind at all, really, and trying to make that work with continuous integration then becomes difficult because you cannot really have builds typically that are, you know, 4 hours long. Any advice really for stopping that from just **4:58** build pipeline from perpetuating the **5:05**, allowing developers to push things further down the pipeline?

JEZ: Yeah, I mean, that’s actually right; I remember you pasted a blog based on this a couple of years ago, and it is an excellent point. Because, there is a tension, right? On one hand, you want a check-in suite that runs in under 10 minutes; on the other hand, you want really comprehensive automated acceptance tests to prove that, you know, the latest change really does work, and you have not broken anything. So, it is tricky, and as you point out, there is a tendency for the developers to pay attention to the check-in suite, and to ignore the acceptance test suite, and then you just end up with acceptance tests that are red all the time until you come to release, and then suddenly there is a flurry of activity to try to get in green, and half of them are thrown away, and then you are back to the initial problem that continuous integration is supposed to address, which is the fact that you spend ages integrating your code right at the end. So, yeah, I mean, it is a big problem. One of the things, and actually Cruise, one of our design constraints was to try and make that easier. So, one of the things that we make it very simple to do in Cruise is to throw resources at that problem. So, taking a log-running automated functional test and making it run across a grid of computers is very simple in Cruise. So, Cruise has the concept of stages, which are like stages in your build pipeline, and you can define many, many jobs in those stages, and hence, sign a part of your automated functional tests to each of those jobs, and run them in parallel on the build grid. So, what we do do is to make it very simple to make those test suites run very, very fast, by parallelizing them across the build grids. So, that is the number 1 solution. So, for example, Mingle has this automated acceptance test suite that takes 13 hours to run end-to-end. We have a grid of 60 computers sitting in Beijing, and when the functional tests run on those simultaneously, you get your results back in 45 minutes.


EZ: So that is the kind of thing that is very simple to do within Cruise. I mean, it is still the case that developers can ignore acceptance tests. In a way, I don’t thing there is a technological problem to that. To some extent, product managers have to just, you know, give developers a bit of a kick, and say actually, you know, it is not done until the acceptance tests are done, you know, and you cannot sign off a story until the acceptance tests agree, which is hard to do in mid-flight. It is easier to do if you enforce it from the beginning. So, I mean, tools are not going to fix all your process problems, absolutely, but I think Cruise does make it easier to get fast feedback.

JULIAN: Sure, at the end of the day, it is about discipline, and this is an industry not always well-known for having it. So, that is all the questions that I have. Thank you for taking the time out to answer them, and is there anything else you wanted to say before we stop recording?

JEZ: Um, no; I think that, well, I guess the only think that I would say is that 2.0 is in the works; one of the kind of sneak previous I can exclusively give you is that we are redesigning the **7:58**. We did not give a lot of love to the **8:03** 1.0, and we are tightly re-doing that for 2.0. So, that is quite exciting, and hopefully people will like that and it will make it a lot easier to use. And, 2.0 is due out, and hopefully there is going to be an early access for people to check out in and around the May time-frame. So, I would encourage people to check that out, and obviously we will blog about it when it happens. And yeah, otherwise, it has been a pleasure talking to you. Thank you very much.

JULIAN: All right, thanks Jez


Continuous Interviews: Jez Humble, Cruise

Cruise is the new kid in the block in the Continuous Integration space. So what’s going on? What can we expect next from ThoughtWorks Studios? In this two-part interview I attempt to find out. Part two will be out next Tuesday.

Jez Humble is the product manager for Cruise. I’ve known Jez for a long time. He’s the affable one in the yellow. I’m the one in black with with the annoying accent.

PS: Jez is writing a book on building and deploying software. I’ll be bugging him for a review copy.

Update: here’s the transcription:

JULIAN: Today, I am talking with Jez Humble from ThoughtWorks Studios. Jez is the product manager; is that right, Jez?

JEZ: That’s right.

JULIAN: For Cruise, the continuous integration product from ThoughtWorks. So, really, I will start by getting Jez to say a little bit more about who he is, and what he really does.

JEZ: Right. So, like you said, I am product manager for Cruise. All that means in practice is that I do a little bit of everything. In theory, I am supposed to kind of define what Cruise does, and look at the market, see what the competition is, and generally own the vision for Cruise and where we are going with it. In practice, I end up doing a bit of that, a bit of analysis, some sales, some support, and occasionally a bit of development, much to the consternation of the rest of the development team.

JULIAN: I get similar accusations with my team. So, can you tell us a little about Cruise, and what actually makes it special as a product.

JEZ: Right, so, Cruise is not just a continuos integration server, is the idea; I mean, there are plenty of CI servers on the market. We are also trying to do something in the release management space. So, it is not just aimed to developers and development teams; it is also aimed at QAs and operations people, and the idea is that you are getting software from [ED: lost the video quality here]. It’s about getting software through the last mile, not just, you know, QA passed, but live, working, released, in production, and that is really, kind of, the source of many of the distinctive features of Cruise, like the release pipeline, the deployment pipeline functionality.

JULIAN: All right, cool. And, um, how long have you been actually using the product yourselves; I mean, there must have been a point at which you managed to switch off of CruiseControl, I guess, and actually eat your own dog food. How long have you been doing that now?

JEZ: For quite some time. So, we started developing Cruise back in December of 2007, and in 2008, by spring festival, Chinese Spring Festival in 2008, which was in March, I believe, we were self-hosting on Cruise, and we had an internal ThoughtWorks project hosted on Cruise as well, which is one of our first customers. So, we are very serious about dogfooding, and we try and make sure that we have projects, within ThoughtWorks, dogfooding as well. So, for example, Mingle uses Cruise in order to build Mingle. So, yeah, I mean, it took us about 3, 3 or 4 months, to be self-hosting with Cruise, and we have been using it ever since.

JULIAN: That is pretty sweet. I mean, 3 months to actually getting a usable product; that is kind of nice.

JEZ: Yeah, I would not go as far as usable, but it got usable pretty quickly.

JULIAN: Very good. I think, as one of the things, you look at some of the other products, and you can tell the ones that are extensively used by the developers and the ones that aren’t, and other little features you see I think are, um, the kind of thing that you would never actually think to put on a product sheet, but the kind of thing that just evolves as your developers get annoyed with the product, or not.

JEZ: Yeah.

JULIAN: And it affects it. So, it is a lovely sign for a product, when it does that.

JEZ: Yeah.

JULIAN: So, I think you really touched on this, that Cruise does not just do CI; I mean, do you think that the CI market is going to support all the venues that we have right now? I mean, you have got the economic downturn; even though I really don’t want to mention that because everybody else is, and also, sort of free competition as well from increasingly more impressive things like Hudson. What are your thoughts on that?

JEZ: Yeah, I mean, it’s true, though. We do have a lot of competitors out there. I think, actually, to be honest, it is not something that worries me tremendously, partly because I think we are not actually that far up the curve of adoption of continuous integration. So, I think there are a lot of developers out there, and a lot of development shops that are not using any kind of CI at all right now. So, I think there is a lot of space for expansion to enter that market. I also think that, you know, we are not really necessarily direct competitors with the Hudson’s of this world. I mean, yes, Cruise is a CI server, but it is trying to do something a bit more than Hudson in terms of the pipe-lining, especially that whole piece around release management, actually deploying, testing, and releasing of software. So, in many ways, we compliment tools like Hudson. I mean, if you have Hudson already, that is typically very much focused on development teams; it is not focused at helping QA teams and operations people. So, we can kind of compliment what Hudson does in many ways, and actually, I don’t think there are that many products which were really aiming at solving that whole problem, all the way through to release, in the same way that we are.

JULIAN: Sure, I mean, um, in my day job, I come across that problem all the time, and I think that is interesting that you are able to say that the 2 products compliment each other. Will there be some kind of direct integration at some point, do you think? Or, will it always be kind of a hands-off thing, where the 2 products just happen to work on pieces of the same work-flow?

JEZ: I would actually like to see some direct integration. I mean, I do not think we will ever have a Hudson plug-in, but I think what we would have is some kind of integration piece where you could take artifacts that are produced by some other system, and then visualize them being released all the way through to production. Something like that, where you can plug into something which produces, kind of, artifacts of one kind or another, and, in fact, that is the kind of direction which we are going with Cruise, anyway, of being able to take artifacts of one kind, and put them through this kind of pipeline process.

JULIAN: All right, lovely. And, do you think the continuous integration market is going to, sort of, get a bit of a boost with every business sort of focusing on cutting waste at the moment? Do you think there will be a direct kind of correlation between that? I would like to think so, but um…

JEZ: Yeah, I mean, I like to think so too.

JULIAN: The cynical part of me thinks that we will do other mad things instead.

JEZ: That’s…

JULIAN: Are you able to… Go on.

JEZ: All right. Obviously, I would like to think that we are going to make money as well; because, otherwise, we are all in big trouble. So, there is, kind of, the theory, and there is the evidence. I mean, in theoretical terms, I think it does make sense, actually, to start investing in these products. What you see is a lot of companies that are starting to lay off people, unfortunately. The result of that is that people are having to do more with l
ess; so, you know, you cannot have a couple of developers kind of posturing around, hacking up CI service in the same way that you might have been able to before. So, it actually makes more sense to invest in a product than it might have done when there were less jobs on the line. So, it is an unfortunate consequence, but obviously, it is kind of good for us, and we have seen that reflected in sales. I know it has been tough, but it has not been nearly as bad as we were expecting it to be; there is still a decent pipeline of sales for Studios products. So, as far as we are seeing, actually, our play, and, you know, it has always been our play, as you know, having been at ThoughtWorks, is that we do the whole Lean thing, and that should help make businesses more profitable and more efficient, and that play seems to be working for us at the moment.

JULIAN: Okay, very good. So, are you willing to say who is using Cruise?

JEZ: Right, uh…

JULIAN: I mean, sort of, do you have a reference site or two, or some names you can drop that you have there?

JEZ: Unfortunately, so I did check on this before I joined the chat; I am not allowed to say who are our customers at the moment, but you will be seeing some case studies in the next few months, as we move towards 2.0. I can tell you, we have gotten more than 3,000 downloads so far. So, we launched in July of last year. So, being going for about 9 months now, we have had 3,000 downloads since then. We do have some Fortune 500 companies in there, and some big name brands, but I can’t tell you specifics yet, sorry.

JULIAN: Okay, cool; you have to try. So, I mean, ThoughtWorks has been incredibly prolific in pushing the continuous integration space for a decade now, and you have, you know, the 3 main flavors of Cruise Control now; the Java, the .NET and the RubyVision, as well as a few things that fell along the wayside. What is ThoughtWorks’ interest in those now; is that kind of just something for your engineers to get involved with as kind of a part-time occupation now, and Cruise is now the main, sort of, focus of your innovation?

JEZ: That is an interesting question; certainly, all those 3 products are still in use, and they are still very active products, projects I should say. Even within ThoughtWorks there are projects that use those things. It is interesting working for ThoughtWorks, because actually, one of our most difficult customers are other thoughtworkers, who are very sensitive about what they use, and rightly so. I mean, as a consultant, you have a duty to choose the best tool for the job at hand, and that might not always be Cruise. I mean, Cruise is only 9 months old; it is only at version 1.2. So, we do not have all the things we want there yet, and there are circumstances in which we might say, Well, actually, you know, maybe you should use or maybe you should use one of our competitors, indeed. So, that does happen, and that is great feedback for us. The reality is that all of those open source projects are still in use, we still have committers at ThoughtWorks on those projects, and they continue to do a great job. ThoughtWorks does not actively put money into those products; it provides infrastructure for some of them. I am pretty sure that we might be putting very much, in the way of money, into paying for developers to work on those products, but they are certainly still in use. Also, I would like us to get to the stage where Cruise is sufficiently better than anything else, that you wouldn’t consider using them, but we are not there yet.


Continuous Integration Cage Fight – the entire first series

CITCON Europe 2008. Tom Sulston proposes a session for CI developers and vendors to show off their wares. The Continuous Integration Cage Fight is born. Seven geeks. One prize of … nothing.

Your humble narrator was given the job of recording the battle, which he did by juggling an Asus Eeepc, and a Nokia N73. Given all that, it’s not a shock that the video quality isn’t good. For CITCON Europe 2009 (registration is open) he’s packing a Flip Ultra.

This is a list of all the write-ups, with video. The fighters were:

The winner was .. everyone. You’re all using CI. Think of those poor souls who aren’t.

Image by


Continuous Integration Cage Fight: Buildforge

This is the final (and very late) installment in the CI cage fight series. The last speaker was Leuwie from IBM, discussing IBM Rational Build Forge.

By this stage in the talk I was done with trying to take notes (curse you, cute eeepc that caught Tom’s eye!) and film at the same time. Things got a little sketchy. Here’s what I managed to note down:

The licensing model is by concurrent user. As it truly is an enterprise product, you can’t just go quoting fixed prices ( as different clients have all sorts of deals for buying their gear). We were a tough audience in that respect, as most of us come from a small product/small team background.

BuildForge understands environments. My notes said ‘environments are mapped’: that was probably something profound six months ago (on the bright side, it’s less than six months to the next CITCON!).

Build agents have manifests that describe their capabilities. Pretty standard. It will parse build output and fail on if a given string is detected. It also supports LDAP. Good.

BuildForge runs commands as lowest common denominator for integration. That includes VCS access.

It has wide platform support. I think Leuwie mentioned something about Nintendo support as well – which means NetBSD support. Officially, it supports:

Windows, AIX, Solaris, HP-UX, Linux, Mac OS X, z/OS, i5/OS

This is possibly the best agent support that I’ve ever seen in a CI server and that’s what I’d say a uniqe feature is. If you’re already drinking the IBM Kool-aid, then it’s probably a very good fit for you. Which brings me to my main point; I’m probably not the guy to be commenting on this stuff.

I might have worked with WebSphere, DB2 and AIX. I’ve even installed OS/2. Still don’t really understand the culture – even though I apprecate what IBM’s research has brought to our industry. So all I can do is say thanks to Leuwie for fighting in the cage, and a huge thanks to IBM Nederland for hosting a great weekend in Amsterdam.


Every Continuous Integration server that supports Ruby + Git

As I posted, I’m on a quest for the right continuous integration server for my day job. I had some helpful suggestions in the comments (thanks guys). I thought I’d maintain the whole list here (in no particular order):

If I’ve missed any other continuous integration tools, please let me know. Thanks.

Update: Added Bamboo. Thanks Ken.
Update(August ’09): Added Ci Joe.



Tagged , ,

Finding view logic in your application

We all know that mixing business logic and presentation is just plain wrong. If you don’t believe me, go have a look at a classic ASP or JSP application of yesteryear. So the prevailing opinion is that we should seek to move business logic into the back end – in MVC applications, the controller (mmm – not always ideal) or the model (perfect).

My colleague Alan and I were discussing this issue – in particular how metric_fu doesn’t inspect our view logic. We have a solution. Here’s the alpha version of our new tool, metric_view_fu. It finds logic buried in HAML view code.

egrep -cir '- if|- else|- while' app/views/* | sort -n -k 2 -t ':'

I think there’s room for improvement.


The quest for a decent Ruby Continuous Integration tool

Git has become very popular in the Ruby community. Github in particular has become a focal point for Ruby innovation. So which Continuous Integration tools support Git and Rake (the Ruby community’s build tool of choice)? I’ve been trying a few out for the 1Click2Fame build process.

What I am looking for is a Continuous Integration server that:

  • supports Rake
  • supports Git, via Github
  • can run multiple agents (I want to test our website against many operating systems)
  • can stop a running build (sometimes, you know it’s not going to work out)
  • has a configuration gui

First out the gate is the daddy of Continuous Integration tools, CruiseControl. It supports both git and rake out of the box. Good.


  • It works
  • I know it very well


  • It works on a single node
  • You can’t stop a build
  • No GUI (unless you go visit Al Wick)
  • The new Java GUI doesn’t render Ruby output very well. The XSL approach may have been as old as the hills, but people did tinker with it.

Next out is CruiseControl.rb. This project was sponsored by ThoughtWorks as a way to make it easier for Ruby projects to use Continuous Integration. I kind of hate it. The original CruiseControl ended up releasing Rake support before the Ruby port happened.

Anyway, it’s done. There used to be this meme that you had to have a CI server written in the language that your engineers wrote every day. I don’t think that’s the case now. These days, I’d suggest that you could do something cool with JRuby to not write all that code yet again (ThoughtWorks were forced to port CruiseControl to .NET by a customer who actually banned Java code from their site – not at all insecure then).

It supports both git and rake, but it’s always felt a little fiddly to me. You have to add a project from the shell on the machine, not from the GUI. The Git support seems like an aftermarket accessory rather than a factory option. Here’s how it stacks up:


  • It works
  • Ruby hackers can hack it
  • It does render rake output nicely, until you reach a limit, and then it truncates the output. You can see the full raw output.


  • Single node
  • Can’t stop a build
  • No Config GUI

While CC.rb works, it’s not rocking our world. The quest for Continuous Integration greatness continues. Next episode: Cruise.

Don’t be disco, use sudo

Interesting comment from Ken Mayer on my post about root passwords:

No one should ever use “root” for anything except single-user mode emergencies and initial configuration. Make it a long string of random characters and store it in a safe or encrypted on a secure hard drive. Make it unique for every box. Then forget about it.

I didn’t frame my post very clearly at all. In some organisations, it’s appropriate for people to know root passwords. Most of the time, you shouldn’t use them. You should use sudo. Why?

Sudo is your best friend.

Sudo allows you to grant access to ordinary users, without having hand out passwords. Passwords are a good start for establishing identity (authentication), but not so good for controlling access to shared resources (like the root account of your server). Something is only a secret if one person knows it. If two people know it, there’s no secret. Three, and your mom knows.

Using sudo to allow your systems administrators or otherwise anointed people to gain access to the root prompt (or even better, just run the commands they need to) means that they each keep their own secret – their own password, which they use to inform the sudo command that they are who they say they are.

I’ve been using sudo since I first compiled it in 1999. I don’t even remember the root password of my main server. I’ve also broken the config badly enough that I have locked myself out of some systems, so I don’t recommend this strategy.

Sudo is now built into Linux, and Mac OS X. Sun distribute it for Solaris, but not in the default install. Solaris ships with RBAC, which is their own very fine grained version of sudo. Sorry, Windows users. You don’t have much more than ‘Run As’, as far as I know.

So if you’re using still using the root password every day, you’re being a little Disco. Sudo was written in 1980. But it’s time is now.