Monthly Archives: June 2008

Psake: new PowerShell based build tool for .NET

Psake: new PowerShell based build tool for .NET

(image taken from mx5tx’s photostream)
The news is pretty much what the title says: A guy called James Kovacs has implemented an experimental make clone in under 200 lines of PowerShell. You might think that it’s another doomed build tool. Personally I think this one might have legs. Only time will tell. But I think it’s interesting for two reasons: PowerShell itself, and the approach.

Rake: The approach of implementing a build tool in your favorite dynamic language by using blocks or closures was demonstrated by Jim Weirich, the author of Rake. The first working prototype took Jim about an hour:

… would need to register the tasks, you need some way of specifying dependencies between tasks, and some way of kicking off the process. Hey! What if we did … and fifteen minutes later I had a working prototype of Ruby make, complete with dependencies and actions.

(from the Rake documentation)

Compare that to writing an XML based tool like NAnt: parsing the XML based build file is enough work, let alone implementing anything else like dependency resolution. Psake in it’s present form is under 200 lines of code, and my Ruby 1.8 install of Rake has about 1300 lines of code, with a bunch of new tasks:

Fox:/usr/lib/ruby/site_ruby/1.8/rake jsimpson$ find . -type f -exec cat {} ; | wc -l
1271
Fox:/usr/lib/ruby/site_ruby/1.8/rake jsimpson$

NAnt, by comparison has considerably more. Just one of the files that deals with parsing XML, without tasks is bigger than Rake:

Fox:~/Desktop/nant-0.85/src/NAnt.Core jsimpson$ wc -l Element.cs
1683 Element.cs

Now: size isn’t everything, and this post isn’t a NAnt bashing post. I’m just trying to illustrate one of the reasons why I think that this tool might go places: it’s a very elegant and simple approach that relies on someone else having done 5 years or so of development.

There could be some merit in exposing NAnt or MSBuild tasks in Rake via Iron Ruby. But someone who already uses PowerShell will probably head in that direction.

PowerShell:

Disney and Apple/Microsoft are in the same business: short-circuiting laborious, explicit verbal communication with expensively designed interfaces.

(Neal Stevenson, In the Beginning was the Command Line)

Does anybody else think it’s interesting that the other Disneyland of operating systems is slowly making everything scriptable? There’s a growing number of Windows applications that you can manage, (interactively or not) from PowerShell. Which could be a huge step above the polyglot of C#, VB, and batch files that people seem to use right now. If PowerShell gains enough ground, it might not make Windows administration so damn hard.

Anyway: check out Psake (and no, it doesn’t rhyme with ‘make’).

Link (via email from Ian Cooper)

Tagged

Ant Best Practices: Use Ant as the Least Common Denominator

We’re back to the best practices this weekend with 12 of 15: Use Ant as the Least Common Denominator. What are we talking about? The answer is here.

What does the common denominator mean? There’s generally a conflict around this on software projects. I’ll explain:

The developer wants to write code. Fair enough. That’s kind of their job. IDE’s give them an immense productivity boost in doing that job. If she needs to switch to another application, the developer will lose focus. So there’s a booming IDE plugin market. So developers will tend to avoid vendor tools in favour of IDE plugins: version control plugins, database plugins, tracking system plugins, etc. By doing so they can become very, very effective.

The release manager and his superiors have a different angle on this: they want to know that the code is deployable. To be deployable, you have to be able to build it. So they have Ant, or some other build tool. That way they know that the code will consistently build and pass unit tests.

Developers (like most people) don’t tend to like impediments to their productivity. Release managers aren’t fond of code that they can’t metamorphose into a working system whenever they feel like it. So developers would rather build all the code using an IDE than have to switch to a command line and build the code. Most release managers don’t want a dependency on a tool that doesn’t really address their needs. It can also be difficult to automate IDE builds.

So the conflict is that each camp has valid reasons for not liking the tools of the other. Which way do you tip the scales? Projects that aren’t constrained by developer productivity (i.e. maybe you have a shortage of testers) should be setting clear guidelines about the build tool. On the other hand, I’d get the kid gloves out if the developers were under pressure to deliver. Eric M. Burke suggests that you at least make sure that:

  • There’s an Ant build that the developers can use
  • They run it before checking in
  • They can use whatever tool they like while until checkin time.

That way, you know that you can reproduce the software later; that Bob didn’t check in code that builds against the development version of a library. The regular checkpoint of the pre-checkin Ant build will allow some flexibility for the developers.

It works for me.

Tagged ,

iPhone build status application for CruiseControl

Alex Hung from ThoughtWorks has written this nifty app so you can check CruiseControl build status on your iPhone. Putting aside all concerns like work/life balance, it’s pretty cool.

Link

Tagged

The guardian.co.uk Build Radiator – a guest post by Michael Brunton-Spall

The guardian.co.uk Build Radiator – a guest post by Michael Brunton-Spall
Michael Brunton-Spall and I both worked on the Guardian ‘R2’ project – in fact, Michael is still there. Last year, we worked together a couple of times on that project. In this guest post he discusses the thing I’m most pleased that we worked on together – an Extreme Feedback device known at the Guardian as the Build Radiator.

Q: Who are you?

Hi there. I’m Michael: developer, Guardianista and build radiator fan.


Q: So what is a build radiator?

A build radiator is anything that is able to radiate information about the current status of the build to all the team members. Examples might include lava lamps, strings of fairy lights or as in our case, a computer with a large display attached to it.


Q: Surely developers know what’s going on with their builds?

Developers are the only people to have ccTray, whereas a big TV on the wall is visible to everybody on the project. CCtray also provides limited information: is the build Red or Green? Our TV is able to display a lot more analog information.


Q: Okay. Why would other people in the team want to know about the Cruise build?

Of course business owners want to see whether the build is green, building or broken. If they see it’s broken a lot, they get feedback that something is wrong with procedures. The people preparing to launch the software want to see what the latest build is doing. But our system does more than that: QAs want to see which builds are deployed to which environments. Everybody should be interested in the live status of the production system, and so on.

Q: I think I get it. What does your radiator display?


We have a number of modules, and the display rotates between different modules. Firstly it shows the always shows the status of different builds, both trunk and release builds.
It also displays:

  • a live view of the guardian.co.uk front page,
  • a list of what software build version is deployed to which QA environment
  • a ‘message of the day’ pulled from a Wiki page that anyone can edit
  • a graph of build duration from CruiseControl for the last few days
  • current status of our live servers
  • a log of the last 5 things said in our project IRC room.


Q: So how did you go about writing this radiator?

Firstly Julian, one of our ThoughtWorkers at the time, wrote a spike. For those of you unfamiliar with XP lingo, that is an ugly hacked together proof of concept (Julian’s words, not mine!)


Once the concept was approved we scheduled a pair from our project to develop the code. We spoke with the product owners. For a radiator, the product owners were the lead business analyst, the technical architect and the team leads. We got ideas from each stakeholder as to what they wished could be shown if we had an infinite amount of time. We then got all the stakeholders to agree a priority order for the tasks.

The requirements of the project were to have a display that: updated fairly often; was visible from across the room and radiated that information out to people, so was visible in their peripheral vision. We also had some specific information requests, things that people wanted to know, examples included a customisable message, a list of currently deployed versions and the build status indicators themselves.

[Julian: We had a request for live cricket scores, which was sadly denied]

Q: Isn’t that a lot of work for non-production code?

You might think so, but actually, we spent only about 2 or 3 days of development time on the first iteration. When the amazingly successful results could be seen, the team managers could see that the benefits were worth it.

Think about it: we have approximately 80 people on our team. If we save them 15 minutes a day each in finding information, then we made back approximately 20 man-hours of development in the first day of use alone, if we even use the conservative figure of saving the developers only 5 minutes each a day, we still save 6 hours 40 minutes of man-hours a day of running.

Q: What did you implement it in?

We decided to write it in Ruby. I can’t remember the exact reason why, I know that we wanted a fast to develop and deploy language, and I think we knew that with a limited amount of time, we had to get as much value as possible. This was my first Ruby on Rails project, and I found Ruby to be very easy to pickup and run with.

Q: Yes, but will it scale?

Why do you need it to scale? People talk about the scalability of Rails, but in terms of a build radiator, it needs to run on a single machine, 24 hours a day. We don’t have several million visitors to its website every day. If we did, we may have to rethink the way that we did it. To be honest, our code would probably scale to some degree. We used the rails methodology of decoupling the views from the models (mostly), so for many modules the number of viewers doesn’t matter. Some models get information from our servers on request, which would overload our backend servers (especially cruise control, the poor thing) easily. But we can cache that if it ever becomes a problem.

Q: Isn’t Rails a web app framework? What made the image on the TV?

That’s an interesting question. We got a PC, connected to the TV, running Ubuntu linux. It runs the Rails server locally, but also runs a fullscreen version of Firefox. We figured that it was the easiest way of displaying the data we needed without writing our own fullscreen application.

Q: What happened after the first iteration?

Well, because we arranged to get the hardware before we started, the developers in the room noticed what we had started, and started coming up with ideas of their own. We had a large list of possible ideas, which we put on hold for a month or two. The stakeholders decided that a second iteration was needed after a few issues that we could have spotted if we had implemented some of the ideas. So we went back to work, and added a graph of current build times, a basic burndown chart and the facility to display some page rendering times of the guardian.co.uk site.

At this point we needed for the first time to be able to store data. Previously the
app was stateless: it queried the data it needed as it needed it. So we had to add the Rails database and model work a bit more. We also got some help from a rails developer who refactored the code into the models from our overweight controllers (our bad!)

Q: What did the team think? Was it a success?

Unimaginably so. The first message was updated within 10 minutes of going live to let people know that a development environment was going down for an upgrade. These days developers don’t question the radiator, it’s just part of the furniture. If it was gone, many of our processes would not work. In daily standups, people often refer to the current build, or deployed version as per the radiator. I see developers turning to look at the radiator all the time, and the facility to see the live status of the servers is very helpful in ensuring that developers notice when the site goes down. We’ve had requests by developers, QA’s, business analysts and our systems operations team to add things to the radiator.

Q: What are your favourite bits of your build radiator code?

I think the decision to display the build status at all times, and the other page in a rotating iframe beside the build status. The build status is used so much, and is so important that it must be on every page. Using an iframe to show the other module has meant that the list of the modules is simply a list of urls, mostly relative. But that has meant that adding a new module to display can be added by either adding a new rails module, or a remote URL, allowing us to tie in with other development efforts such as the page rendering stats, or a server status perl script that our operations team has.

Q: What is your least favourite bit of your build radiator code?

As a development shop we develop in Java, using Spring, Hibernate and Velocity as core technologies, we don’t use ruby or rails for anything inhouse. We are lucky to have ThoughtWorks as a consultancy, so we have a few Ruby programmers around, but most of our guardian.co.uk programmers don’t do Ruby. It would have made more sense to write it in Java, with Spring, and we could have used it as an opportunity to try out some features of the new Spring release, or to try JSP instead of our present view language. However, I think that the time taken to get the project started would have been at least a week instead of two days. Whether the technical leads would have gone for that amount of time without seeing the benefit? Who can say?

Michael, thank you. It was a pleasure.

Tagged

Subversion 1.5 has been released

With shiny new features. The nicest feature for me is Merge Tracking, which means you don’t have to manually keep track of what’s been merged from your release branches to your trunk.

Link (via announce@subversion.tigris.org)

Build doctor: a standalone blog once more.

Build doctor: a standalone blog once more(image arranged and photographed by my 5 year old daughter)

Its been a little over a week since this blog was removed from the Planet TW aggregator. It was absolutely the right thing to do. That didn’t stop me being nervous about doing it. The problem was the lack of feedback from the users that read the blog via Planet TW’s website or RSS feed. Darren Rowse of problogger.net had kindly confirmed my suspicion that I wasn’t really connecting with people.

The other issue that I had was that none of the branding came through Planet TW. It was all posted under my name and not the Build Doctor brand.

After some correspondance with the ThoughtWorks IS department, Last Thursday evening Hongtao pulled this blog from the aggregator for me. And nothing really happened. Well, I went to the pub (my local has Broadside on tap!).

Hits still continued to be mainly from google searches, with a few referrals and RSS readers. Traffic was a little down at the beginning of last week but:

  • I hadn’t posted much content last week anyhow; and
  • The drop wasn’t all that significant, and things picked up a little.

What has been interesting is the growth in subscriber numbers. Build-doctor.com is a new-ish, niche blog that I haven’t promoted very much. The subscriber numbers haven’t been huge. Since the disconnection, the RSS subscriber numbers have been slowly but steadily growing.

Hurrah. A metric that I can actually use.

CruiseControl radiator – very disco

CruiseControl radiator – very disco
Originally uploaded by Znachor

It looks like something that my Dad would have put together in the 80’s. I like it.

The problem with Lava lamps is the granularity. Great for your 6 person project, not so great for the larger team. I manage around 80 different CI builds at work. A Bettabrite scrolling LED sign would be better for us, I think. Not sure I could get away with the retro look in our office.

Tagged

Hudson: now in a Debian package near you

Hudson: now in a Debian package near you
(image taken from oskay’s photostream)

Packages are your friend. Really. If you’ve done systems admin for a while, you’ll realise that it’s worth the effort to do installations via a package. There’s 2 main benefits to my mind:

  • You can easily remove the software later, upgrade it, or both
  • In some systems you can also get checksums for the installed files – on Solaris systems for example you can run the ‘pkgchk’ command and verify that all of your software is intact.

So I’m quite pleased to see that the Hudson project now builds Debian packages. Nice work, people. I might get around to evaluating it properly now.

Link

Tagged

Ant Best Practices: Use version control

Amazingly, it’s article 11 of 15 in my series on Ant Best Practices. Today’s practice is ‘Use version control’ and I can’t help but wonder if this one hasn’t dated a little. When the original article was written, Subversion didn’t exist, Perforce wasn’t free, and most people used CVS. I’m not going to get nostalgic about CVS. It was better than the alternatives (ever used SCCS?). I wouldn’t go out of my way to use it now.

So would anybody actually challenge this anymore? The original article by Eric M Burke suggests that people would version code but not the build files. I’d be gobsmacked if anybody did that nowadays. Drop me a comment or note at ‘medic@build-doctor.com’ if you seriously disagree with version control for your build files. I’d really like to hear why.

I’m chapter and verse with Eric here:

  • Version control your code
  • Version control your build files
  • Version control your binary dependencies (unless you have an external dependency manager). Don’t version control build output.
  • And try not to mess around with checking things in and out of version control from your build. It generally gets ugly.

The sin of not using version control has done to death anyhow. Ask Joel.

Tagged

ant logfile /home/cruise/yourgreatproject/log.xml does not exist

ant logfile /home/cruise/yourgreatproject/log.xml does not exist

(image taken from losiek’s photostream)

Welcome to the first post of the Build Doctor that isn’t in the Planet TW feed. If you’re reading this via RSS: thank you. Thank you for subscribing. The point of this exercise was to actually know who my readers are. And now I know.

Back to the post. I’ve been helping someone set up Ant and CruiseControl. He mailed me last night to say that he got a log message saying that that the log.xml file didn’t exist. The short answer is that the Ant build failed.

The longer answer is that CruiseControl calls Ant with an argument to make it use the XmlLogger. Then CruiseControl can hoover up the XML file, and display the output in the Dashboard. What appears to happen is that if your build fails very quickly, the log file isn’t written. So all CruiseControl can do is complain about the missing log file. It all makes sense when you read the code:

if (!file.exists()) {                
throw new
CruiseControlException("ant logfile " + file.getAbsolutePath() + " does not exist.");
} else if (file.length() == 0) {
throw new CruiseControlException("ant logfile " + file.getAbsolutePath()
+ " is empty. Your build probably failed. Check your CruiseControl logs.");
}

Of course, there’s other reasons why this could fail, but every time I get this message, I find out that it’s down to a very broken build or broken Ant install. I’m tempted to try and reproduce this issue with the XmlLogger.

Tagged ,