Tag Archives: hudson

Hudson: trouble at the mill?

Looks like Oracle and the Hudson developers aren’t getting along. The project hosting that Oracle provide (inherited from Sun) has been a bumpy ride for the developers, who are suggesting GitHub as a less bumpy and more functional way to develop. Oracle seem reluctant to let them go. Which you can understand, in a way.

Summary at hudson-labs.org, full story on the mailing lists.


Hudson hit my Puppet with Cucumber

I felt I had to attend the next talk in the main lecture room, simply to find out what its title meant. Patrick Debois and Julian Simpson’s presentation was entitled Hudson hit my Puppet with a Cucumber …

I think that’s some of our best work. (via UKUUG June newsletter)

Tagged , ,

Tool News, February 2010

There’s a new Hudson out. It’s got 7 bug fixes and 3 enhancements, including some profiling and optimisation work to speed up the UI.

Atlassian have embarked on a Build Engineer hiring offensive. They seem to be based in Sydney. Atlassian would appear to be a great place to work.

Electric Cloud have released new versions of ElectricAccelerator (now at version 5) and ElectricCommander (at version 3). The former works with make, MSBuild, scons and the like to parallelize the build. There’s a new feature in Electrify, a private cloud that allows to to parallelize more software production. I think they’re on the money with the private cloud. Cloud services may be the meme du jour, but cloud technologies used internally would be the way to for many companies to start.

Scott Castle:

Electrify is interesting because it’s a seamless way to tap into distributed computing, and it’s not tied to using a specific build tool. If you’ve looked at the various efforts to build Ant tasks in parallel, or to upgrade Maven for parallel execution, they all focus on local-box parallelism, leaving out the tricky task of distribution across hosts. Electrify bridges the gap, converting any parallel task into a distributed task, and it has all the network, caching, and management ‘nice-to-haves’ that homebuilt systems generally lack.

ElectricCommander looks like it aims to be the tool to make build and release managers redundant. And if that means not doing deployments for other people, I’m not bothered. I’m surprised that the tool hasn’t had more coverage as it seems squarely in the same arena as Cruise and Ant Hill Pro. Today’s release brings a lot of customisation: here’s Dax Fahang, (who came to Citcon Paris, some readers may recall):

Developers are often resistant to the introduction of new tools or processes, so we find that customizability is critical to adoption. With ElectricCommander 3.5, users truly have a build-test-deploy system that works and looks exactly the way they want it to without all the traditional effort and costs associated with customization of a homegrown system. Version 3.5 allows customers to customize the UI without any application language or upgrade constraints – thus enabling Commander to easily look like a legacy tool, integrate data from multiple applications, and provide role-based user experiences.

Tagged , ,

The official hudson weblog

R. Tyler Ballance has been unveiled as the mastermind behind the @hudsonci Twitter account.  He’s been working on an official Hudson blog. I’m hoping this will be a stepping stone for building out more Hudson community and conversation. Well done, all.

continuous blog | the official hudson weblog.


Selenium and The Dialog of Doom

This is a guest post from Douglas Squirrel, CTO at youDevise

At youDevise, we’ve been suffering from what Ivan Moore calls Flickering Builds for some time now – the build fails with a false negative once, then passes the next time we run it.

Our Cruise Missile monitor records these failures for us so we can track them, and it lets us mark them so we know not to waste time on further investigation, but it’s still massively annoying when your build isn’t reliable!

One of the worst sources of build flickers is Internet Explorer causing a Selenium test run to hang. When IE has an issue with JavaScript, it throws up a dialog box and then sullenly refuses to budge until it’s clicked. This is particularly bad because it hangs everything else up the food chain to Hudson, and nobody else can run an IE build until we bow to Bill Gates and click OK, dammit!

The root causes of the JavaScript error vary, but the behaviour of IE (when running under HTA mode in Selenium) is a problem that we need a countermeasure for. The Build Doctor made a house call recently and suggested one of these could be the cure for us:

  1. detect the dialog and, when you do, click it or else kill all browsers on the machine
  2. try proxy injection mode for IE under Selenium
  3. modify Selenium to fail the test and kill the browser when the dialog is detected
  4. wait for webdriver to merge with Selenium (it’s supposed to be more resilient in the face of evil dialogs)
  5. wait for IE to die

We’re trying to convince Julian to have a go at one of the first three (webdriver will surely ride to our rescue before the IE zombie falls over, but neither one seems likely before the next Star Trek movie). If we can get it working we’d like to make the solution open source, ideally part of Selenium. Surely others are suffering with the same problem and would benefit from an end to the Dialog of Doom. Stay tuned!

Image thanks to OiMax

Tagged ,

Kohsuke Kawaguchi Hudson Webinar

I thought I was pretty good at guitar once. You only realise how much you have left to learn when you see someone who’s at the next level – and then you realise that they can kick your ass without breaking a sweat. Realising that you’re not as good as you thought you were can have two effects:you double your efforts, or give up in disgust.

Kohsuke Kawaguchi’s webinar on Hudson was an ass-kicking moment. In guitarist terms, he’s a monster. He’s Eddie Van Halen.

Kohsuke Kawaguchi

Kohsuke Kawaguchi

I started live tweeting his webinar once the Java client ground into life on my Macbook, but I switched to an editor when I realised how much depth there was to this. The rest of this post is a fairly linear set of notes on his talk. I’ll be adding links (and maybe a correction or two). Please do comment or tweet if you have anything to add or fix.

18:07 – Webinar client has sprung into life. Kohsuke is already talking about patterns for hudson. Lesson: don’t use hostnames in DNS names. Also, use port 80 for the service. Apache makes a good frontend with mod_proxy.

Sun Hudson server has 600GB of data. You need to back up at least some #Hudson data files. There’s a Hudson backup plugin. @kohsukekawa uses cpio. That’s kicking it old-school. They also recommend ZFS snapshots.

Hudson slave nodes need 1 170KB jar file to run. Single link, with no assumptions about transport. VM’s or recycled hardware can be slave. Do be careful about how you organise Hudson clusters. Be prepared to make more clusters if other people have different requirements.

@kohsukekawa just suggested using the native OS packages that they supply. Fine idea. I do that. Also, use a filesystem that grows well. Again, they use ZFS.

Deploying your operating system is perhaps the best way that you can treat your Hudson slave machines. Kohsuke recommends Windows Deployment Services for Windows slave machines. For pretty much any other OS he’s a big fan of the Hudson PXE plugin. PXE is a standard for booting a computer over Ethernet – Sun hardware always supported this out the box. I used to love booting and installing an operating system from the prompt.

Your other option is to clone Virtual Machines but this is far more painful – how do you version a few gigabytes of operating system? You may be able to get away with building systems by hand, but I’m not fond of the error rate.

Hudson can install lots of dependencies on the slave machine: with a Java Runtime environment and SSHD on your unix systems you can install the Hudson slave on the server you just need the hostname. For Windows, you can pass the administrator username and password via DCOM. It even works from a Hudson master running on Unix.

What if the Hudson master server can’t see the slave machines (ever had an over-zealous firewall administrator)? You can log into the slave machines and use Java Web Start to have the slave talk back to the master. You’re screwed if the slaves can’t see the master.

After you’ve launched a Hudson slave via JNLP you can install it as a Windows Service. The Unix option is to run a headless Java process.

Hudson will also maintain the JDK, Ant, and Maven for you. You need to declare the versions that you need. I hope the sheer enormity of this isn’t lost on some of the readers of this post. That saves so much monkeying around. You can also tell it how to run the installer to get other dependencies. I’d probably tell Hudson how to kick off a Puppet run. Kohsuhke also suggested:

  • Rsync
  • apt-cyg
  • cfengine (the tool that inspired puppet)
  • Ruby devs, you might like to use chef. I still can’t recommend it in production however
  • Windows peeps, you can use Active Directory and the open source WPKG (which needs a reboot)

The downside to puppet/cfengine is Windows. We all need to talk to Windows systems. Hopefully Mr. Nasrat’s work on Puppet for Windows will pay dividends in that respect. Kohsuke recommends Cygwin for deploying apps on Windows.

What should the goals of your Hudson cluster be?

  • Make the slaves a pawn: he used the term interchangeable and nameless. I like pawn. It conjures disposable build slaves for me. The benefit is load balancing within the cluster, reduced false positives, add easier lifecycle management.
  • Depend on labels, not slaves – this improves utilisation. Use a group of slaves in a label so you depend on many hosts, not one.

Hudson is going to put me out of a job. There was a fascinating section on reliability: Hudson will monitor slaves and take them offline if they start running out for disk space, or the clocks aren’t in synch, etc. I believe that he said all these checks came from the field.

Hudson Housekeeping:

  • Use NTP to keep the slaves in synch
  • Keep /tmp clean. i use Puppet for this.
  • Keep adequate records of changes to your cluster (goodness – old school systems admin again!)

Hudson also cleans up the mess after builds. As I understand it, Hudson will attempt to diff the process table before and after the build process, and kill off any processes that are left running. Again, that’s a simple idea that adds a lot to your build process. Of course you want to control the dependencies from your build. If you must have some process running (slow old enterprise software?), the wiki documents how you can switch this off.

Hudson will report load on your cluster. Load tends to even out if you have enough capacity.

Upgrading Hudson is just a matter of upgrading the war file. Do keep the old one. It’s possible to downgrade again. Hudson can also update itself, which I wouldn’t recommend if you use OS packages. You need to update your plugins yourself these days, which wasn’t always the case.

If a release has few subsequent bugfixes, it’s probably a good one.

The easiest way to make builds dependent is to trigger one job after another. You can trigger multiple parallel test jobs after a build, to get faster feedback.

The build promotion plugin allows you to pick up really good builds to promote downstream. It can do this automatically, or from test results. Once promoted, a build can be used further down the development process: to commence QA, deploy, integrate elsewhere, or push to Maven. An example of this: Sun use the promotion plugin for JAXB-RI to push to CVS for delivery to the JAX-WS RI. So good to see these guys dining on dogfood.

Concurrent builds allow you to get faster and faster feedback to the developers, and hence isolation of changes. Good for flickering builds. Use a timeout to stop a bad build exhausting all capacity.

Matrix projects allow you to concurrently build your project on different Operating Systems and middleware. Do use a ‘touchstone’ or ‘canary’ build first. Build them sequentially if you need the same resources (e.g. databases). What if your matrix gives you combinations that are the same, what do you do? You can filter out some combinations or suggest a coverage ratio and let Hudson work it out. Nicely declarative.

18:53 – just lost the audio. Kosuhke’s Conclusion:

  • Planning helps on an enterprise scale Hudson deployment.
  • Throwing lots of resource helps a LOT.

18:55 – audio is back and we’re in Q and A. I chose this moment to go dash for my train.

My conclusions: Hudson offers a huge amount of value for a tiny price. But not just in terms of numbers of features. That gets boring after a while. It’s the thoughtfulness of the features, and the attempts to address lifecycle that have me excited over Hudson tonight. Most other Continuous Integration tools that I have used exhibit a clear bias towards the development end of things. I haven’t seen a tool that reached so far towards the operational side of things. Putting the stack together on the build machines has typically left to the likes of me; now there’s a tool that does that

Don’t tell him, but I was pretty impressed with Kohsuke’s knowledge of systems, as well. I’m sure a lot of these features came from the community, but he has a pretty rounded view. Maybe it’s the Sun Microsystems influence – say what you like about them, but they know good engineering. In any case: we owe this man, the open source team, and Sun Microsystems a lot for pushing the envelope on Continuous Integration.

If I were a Continuous Integration tools vendor, I’d be feeling nervous about getting my guitar out.

Update: did you miss the webinar? Have a look at it here.

Image via Andre

Tagged , ,

Sun enters the Continuous Integration business

Sun sell a Continuous Integration server. We should have seen that one coming. See Kohsuke Kawaguchi’s announcement. Looks like you buy it as an add-on to the GlassFish stack. Pricing is here. Looks like they’ll support a branch of Hudson, and not the open source releases. That’s not unexpected.

Strategically, it seems like a good idea. They’ve been building their own stack for years, from operating system to web server. Sun just pimped it with a very popular Continuous Integration server. I think It’ll be a good proposition for Java shops. There’s a huge market for Hudson for other platforms, but I’m not sure that Sun will be chasing it.

(image by Silveira Neto)


Cruise Interview with ThoughtWorks Studios

TW Studios peeps Chris Read and Andy Yates graciously agreed to have a chat with me about Cruise last Friday. We talked about some of the features in Cruise and why they matter, the difference between Cruise and your average development focussed CI server, and shot the breeze about the maturity of our industry:

It went pretty well, considering just how shattered I was. I managed to edit out the bit where I set off the timer that they use for interviews. Part II on Tuesday, August 4.

Tagged , ,

Continuous Integration Doom and Gloom?

Andrew Binstock has written an article for SD Times, postulating that the Continuous Integration market is rapidly changing; and that the effects of this change will be:

  • Consolidation of the enterprise market to 3 main products
  • Consolidation of the workgroup market to CruiseControl Hudson (as free Continuous Integration servers), and Bamboo (as a paid product)

Maybe. I’m not sure it’s that simple. He’s correct in that marketplace is rapidly evolving, for both free and paid applications. Yes, I think there will be a few dominant products in the Enterprise space, probably along vendor lines. For example, if you’re already invested in RAD and WebSphere, you’ll probably buy BuildForge.

Andrew’s guess is that Team City and Cruise won’t gain critical mass. Team City because there’s no tight integration with the rest of the JetBrains stack, and Cruise because it’s a comparatively late entrant to the market.

But there’s other forces to consider. For one, what about the .NET guys? There’s a terrifying number of them who don’t use Continuous Integration. Will they all go for TFS? CruiseControl.net is looking a little creaky, and there’s been plenty of interest in Team City by some of the .NET luminaries. Culturally, this community can and does pay for software, too.

While Hudson is quickly shaping up to be the alpha male of the CI tribe, it still doesn’t appear to have the installed base of the original CruiseControl. Recently CruiseControl was still beating Hudson in terms of downloads and installed base. If CruiseControl (often referred to as Cruise, which does muddy the waters somewhat) loses any projects to other products, where will they go? My guess is to Hudson if they stick with an open source product, or possibly Cruise, if they like the cut of ThoughtWorks’ jib.

Another trend that will be relevant: the market will be growing for some time to come. It’s increasing in sophistication as Andrew points out; but my guess is that we’re at least a couple of years away from saturation.

The other consideration is the economy. I’d like to think there will be opportunities as well as threats. I have no doubt that there will be less IT projects over the next few years, and certainly some cancellations. But what will the industry’s attitude be to waste? Will it be considered okay to defer integration? Will Continuous Integration be de rigueur? Will development managers insist that we demonstrate some ROI on the CI products that we choose? I hope we get better at this.

Andrew has done a great job in identifying some trends at the macro level. He’s picked some very tasty players for his prediction. But how this will play out? Who will walk away with a chunk of market share? I think it’s still wide open.

I have reached out to a few people for comment and will update this post. The Ant Hill Pro guys seem to agree with the 2-tier analysis of the marketplace.

Update: 13-06-2009 morning – Here’s the response from Ken Olofson of Atlassian:

I saw Andrew’s write up and it’s great that he’s picking us to stick around in a competitive market.. 😉 I think he’s tipping us for the right reasons, too. Bamboo is a core product for us; we don’t just develop integration with our own products, we focus on ease of use, collaboration, and integration with all the tools developers rely on. We try to lead with innovative features by being the first to develop EC2 support and two-way IM integration (still don’t think anyone else is doing this). Much of this comes from our own dogfooding since we use Bamboo extensively across all 12 development teams within Atlassian. Check this out: http://www.atlassian.com/agile/practices/continuous-integration.jsp Oh yeah, and we can’t forget about Atlassian’s legendary support.

He’d be a fool to disagree.

Update: 13-06-2009 evening. Comment from Jeffrey Fredrick, who has been a CruiseControl committer for years, and more recently he’s been working with the Urbancode guys.

I think Andrew makes an accurate distinction between the two markets, and I think my own recent history ads weight to this being more than lip-service. As you know I’ve been working with CruiseControl for a long time, all the way back to 2001. And then I joined Urbancode in April in the role of Technical Evangelist. But one of the points we agreed upon when I joined Urbancode was that I would continue to work on CruiseControl! The reason is that we really do believe that we’re competing in a different space for a different audience with a different value proposition.

Yes AnthillPro (or other Enterprise CI tools) can be used as a classic CI solution. But as teams get more sophisticated they want to automate more and more, and they want tools that can tie together the entire Build Lifecycle. And while some teams do this by writing their system around a workgroup level solution many others want to spend less time maintaining their build & deployment infrastructure and are happy to find an off the shelf solution that fits their needs.

As I just told JTF, I fully appreciate the latter point. People do scary things with build scripts and VCS systems that should be part of the plumbing of your CI system anyhow.

Update: 15-06-2009. Paul Julius pointed out that CruiseControl was tipped to survive as well. Apologies, CruiseControl users: I forgot to put it on the list.

Update: 18-06-2009. Yegor Yarko from Jetbrains responded a couple of days ago. I’ve been remiss. Sorry. Yegor stresses that this is his personal opinion, and not an official JetBrains response:

Andrew’s article is probably a good one to cover business-level, but there is another angle that is more appealing to me.

To my bad (or good) I am not deep into the marketing and would rather judge products based on the feature set/ease of use, etc.

Targeting a product for a specific sector or type of users is a good thing but targeting alone means nothing. I’d rather pay attention to the product attitude and refinement of details. These are the factors that affect the everyday users most.

As an insider for TeamCity, it’s hard for me to make an objective comparison for different CI products. After all, there are different opinions of what is most important or usable and our vision is already implemented in our product.

Different people/companies can value different properties of the products and the classification has probably many more entries then two or three.

From my experience CI process is very individual in any non-trivial size project. These varying requirements give birth to different offerings on the CI solutions market. And while the enterprise/workgroup division is probably correct, there might be key differences between the projects/companies that fall into the same category. No single CI solution is able to cater to all needs.

I’d also agree to your note that the market is yet to grow.

Tagged , , ,

Hudson on Ubuntu with Big Visible results in six steps

We’ve written before about making your Continuous Integration build (and more) available on the big screen. I just told our CEO that a red build meant that we weren’t ready to release code. That’s a key fact to share with the rest of your team.

Yesterday I set this up at work. It was the easiest radiator that I’ve ever done. Here’s how:

Step One: get Hudson on your Ubuntu machine:

sudo su -

echo "deb http://hudson.gotdns.com/debian binary/" > /etc/apt/sources.list.d/hudson

aptitude update

aptitude install -y hudson

Step Two: have a look for the Hudson admin page with a browser. http://${your_build_machine}:8080/

Step Three: set up a simple project. You might need to add plugins at this point to support your projects. I ended up adding the Ruby and Git plugins to support our Rails and Merb projects. Your mileage will almost certainly vary. Don’t worry about making it pass for now.

Step Four: get the build results out there: on the main Hudson page, click the “Manage Hudson” link on the left. Then click the “manage plugins” link. Click the “Available” tab, click on the checkbox next to the “Radiator View Plugin” entry on the plugins and click the “Install” button on the bottom right hand side of the page.

Step 5: make the plugin useful by adding a new view. There’s a ‘+’ button on the hudson main page:

Click that, and make a new view, which you need to map to the plugin. I called mine ‘radiator’.

Step 6: now you need to choose which projects belong on the screen. If you just want them all, then you can use a regular expression to include ’em all. I used ‘.+’, which matches one or more occurrences of any character. Apply that change, and you’re done.

Here’s the radiator at the my day job: