Category Archives: Continuous Integration

SnapCI: everything I ever wanted

Some time ago, I worked at ThoughtWorks.  It was my job to ensure that the CI server was accurately giving feedback to the team,and that we had a remote hope of deploying applications into production.  ThoughtWorks had (and has) some incredible people: I probably should have stuck around longer.

At the time, I was frustrated with the CI tools that we had.  The three different ports of CruiseControl that we tended to use (probably why Jenkins ate CruiseControl’s lunch) all suffered from similar issues:

  •  Configuration wasn’t often easy, especially when dealing with configuration that spanned multiple projects/jobs.  You could write a build-breaking configuration which would break the next commit, through no fault of the developer who committed it.
  • From about 2005 it was clear that this Build Pipeline thing had legs.  But it took a long time until the tools caught up with the Build Pipeline concept.

The industry still has a long way to go to help us fulfil the promises of Agile and Continuous Delivery; but I’m happy to report that my problems are solved, and the tool I’m using to solve it is Snap CI, from my old colleagues at ThoughtWorks.

We’re using it on a few Git based projects at work, and it’s very reliable and simple.  The biggest project has 4 stages:

  • Fast Feedback
  • AMI
  • Staging
  • Production


It shows one real build pipeline for all of those stages.  Any time I make a configuration change to a pipeline or any of it’s stages, the entire pipeline triggers.

Git helps a lot; if I need to know what the current HEAD is, the build scripts can just ask;  each stage has the same access to git.  GitHub integration means that we can control access to Snap via Teams in Github.

 We can store passwords and credentials securely in the pipeline.

 They’re also rolling out Docker support.

The best part of it? I’m not slaving over a hot CI server; I’m working on projects that are useful to my employers instead.  There’s never been a better time to be a developer.

The Godmother of Continuous Integation: An interview with Tara Hernandez

Continuous Integration existed before CruiseControl. Before Martin Fowler wrote about it. Of course, it always was a primarily a practice, but some don’t know that there was a Continuous Integration server that existed before CruiseContol: Mozilla Tinderbox. Tara Hernandez was instrumental in developing the first version of Tinderbox, and I asked her for a little more info:

Q: Where did you start out?
A: Borland was a great place to start out (first real job out of college), and I learned a lot. However, the CEO was insane and there was a strong personality cult around him that made it hard to reign in his bad ideas. For example, Borland was really really well regarded with regards to its compilers and if had stuck to that we might have been okay. But Phillipe Kahn (CEO) had this idea he could beat Microsoft across all technical verticals and so started acquiring other technologies like databases and stuff. We had so many products that we were competing against ourselves in some instances.

Microsoft finally noticed and squished us like a bug. The team at Borland I was on (Languages) got totally pillaged. Microsoft lured away almost all the top talent, including our VP. They probably spent 10 million dollars on signing bonuses. It was a mess.

Anyway, I got hired at Netscape after about 3 years of Borland. Somebody I had previously known at Borland pinged me and told me of an opportunity at this new startup Netscape. I was the first build/release engineer hired, and I only had 3 years of experience at a very sedate well-established company under my belt. Best analogy I can think of was that it was sort of like jumping out of an 747 without a parachute. There wasn’t a consistent build platform, even within the same general operating systems group.

For example, there were two distinct make systems in play in the unix world. The revision control system (CVS) only worked on unix at the time, so for Windows and Mac platforms we had to run this crazy software to export the local hard drives as mounts readable on the unix network, and then people had to run xterm sessions to do any SCM transactions. It sucked.

Q: What was the problem that caused you to need Continuous Integration?

A: It was the fact that we had 2 version of MacOS (68k and PPC), two versions of Windows (16 and 32 bit) and (initially) 8 versions of Unix. At any given time our ability to successfully generate a set of builds was close to nil. Things would work great on one platform, but not on 6 others.

The most famous of these was when Brenden Eich took out the world by checking in the use of a NaN in the early javascript code (not called that then of course, but that’s what LiveScript turned into). Turns out NaN was totally unsupported by 60% of our platforms. Sadness. On the other hand, to show his contriteness he bought me and my team a whole lot of expensive booze, so that worked out okay in the end.

Tinderbox really evolved out of desperate self-defence, because it was almost impossible for us to do our jobs — mainly, get builds out the door in a timely fashion, and let developers know as soon as possible that they broke something somewhere and to please go figure out how to get it fixed.

About Tara:

Tara Hernandez is a specialist in software engineering infrastructure having developed procedures and tools at assorted Silicon Valley companies. Her most notable stint was at Netscape Communications Corporation where she initially helped develop new tools such as Tinderbox, Bugzilla, and Bonsai, and later helped adapt those tools to a public environment after the creation of and the release of the Netscape web browser as open source. Now at Pixar Animation Studios, Tara and her team are reinventing development infrastructure to handle the challenges of writing software in a film production environment.


LondonCI Meeting, 1st November

Continuous Integration matters. Configuration Management matters. What happens when you mix the two?

Let’s find out. Chris Read and I are speaking at the BCS Configuration Management group meeting on November 1. We’ll be doing our talk from Agile 2011, which I hope will spill over into a massive discussion over nibbles and wine.

The venue will be the BCS offices in Southampton Street, London. You need to register for security and catering purposes. Chris and I will be talking from 18:30. We’ll be preceded by the CMSG AGM, which non-members are welcome to attend (though you can’t vote unless you’re a member)


Atlassian Bamboo Challenge

Atlassian kicked off the Bamboo Task Master Challenge today. I am honoured to be on the judging panel.

What’s it all about? A Continuous Integration server becomes the heart of your development infrastructure. CI servers need to know how to talk to the rest of your systems.

So Atlassian are encouraging you to add to the Bamboo ecosystem by offering the winner of the competition:

  • A ticket to next year’s Atlassian Summit
  • $500 of nerd toys
  • A USB Bamboo keyboard and mouse
  • A USB rocket launcher, that integrates with Bamboo

The first 10 entries get an Atlassian T-shirt and the Bamboo Keyboard and Mouse.

I think it’s a clever move. Plugins made Hudson/Jenkins more attractive to it’s users, and gave everyone the incentive to encourage more plugin authors (see Crossing the Chasm for more on this)

Here’s an idea to kick things off: I’d like to see someone use face-detection and face-recognition libraries to detect the face of the person who broke the build.

An targeting system for a Nerf Vulcan wouldn’t go amiss, either.


Repository? That’s not a repository …

this is a repository. I’ve been experimenting with Amazon S3 as a Maven repository. I threw the results up on GitHub [link].

Why? I’ve been working with some nice people who use binary dependencies. In order to scale their CI system past one node, they need a repository manager, to temporarily store built artifacts on.

They actually have a couple of repositories, but not geographically close to where I’m building out their new CI system (on EC2). It’s important to keep the feedback loop fast; so deploying a repository close to(or indeed on) on the CI server is desirable.

The plan was to deploy Archiva onto an EC2 instance via Puppet. Then I had a better idea: use an S3 bucket as a highly scalable, low latency and (mostly) highly reliable repository.

Turns out that there’s support for S3 in their build tool (Maven). So it turned out to be easy. In the github repo, I publish and artifact using an S3 client, and then retrieve it using plain HTTP.

As long as your EC2-hosted Continuous Integration server is in the same availability zone as the S3 bucket, you’re not going to be liable for a high traffic bill.

They may want to use a different configuration for the real repository, where they serve artifacts to the other consumers. This approach is all about speeding up and scaling out their build.

Update: Oliver Lamy asks: ‘what is wrong with Archiva’?

Nothing. I’d cheerfully use it. It has a permissive license and cheerful absence of public spats with other repo managers.

In this case I wanted some very fast and robust, which led me to S3. The alternative was to deploy Archiva on an EC2 instance: I could have done that, but then I would have had a dependency on a single host.

IMO, this approach is more suitable for the context (caching intermediate build artifacts, on AWS).

Tagged ,

#LondonCI, July 20

After a great meeting with Gus Power (watch the video) this month, we’re all set for LondonCI on July 20. David Farley from LMAX will be sharing his experiences with Continuous Deployment. Dave knows a thing or two about the topic, having co-written ‘Continuous Delivery’.

Meetup info here – don’t forget to register at Skills Matter …

#LondonCI Videos

In April, we kicked off with Chris Read.

In May, we had Paul Stack, Andy Parker, Tom Denley, Joe Schmezter [waiting for the video to be processed], and me.

Tomorrow, we’ll be filming Gus Power talking about Continous Deployment in the last mile. Remember to register if you’d like to come …


#LondonCI meetups, June and early July

June’s meetup is this Wednesday, 15th of June (sorry about the short notice) and features Gus Power of Energized Work:

This session will take a look at leveraging continuous integration techniques to deploy and operate software all the way to the end user, exploring some of the difficulties and gotchas along the way.

Gus is a great speaker, with great real world experience in delivering software. He has one of the finest bios I’ve seen in a while:

Gus Power is a hairy force of nature who’s always looking for better ways to figure out what the right stuff is and how do get it done.

Please register on the SkillsMatter website if you want to come. Meetup group info here. Pub afterwards is the Slaughtered Lamb.

For July I’m still trying to secure a speaker but in the interim, why don’t we go to the pub? Andrew Bayer of the Jenkins project is in town and wants to hang out with some Jenkins (or to be honest, any Continuous Integration) users. This will be at the Royal Festival Hall so no need to register with Skills Matter – but registering on the meetup group would help me get a handle on numbers.

#LondonCI Lightning Talks, May 18

It’s almost the third Wednesday of the month, the weather is good, and it’s time to do the second monthly London Continuous Integration meetup at SkillsMatter. After a barnstorming talk from Chris Read, I thought it might be nice to do some Lightning talks – especially as I want to hear new voices talking about Continuous Integration.

Here’s the deal:

  • I’m still looking for a couple more talks – please get in touch if you’d like to do one. It could be a 5-10 minute talk, on any subject related to Continuous Integration, Continuous Delivery, etc.
  • You need to register on the SkillsMatter page so they know what size room to give us. I’m very keen to make it easy for our hosts.
  • If you register on the Meetup Group, that helps me keep you in the loop.
  • Pub afterwards will be The Slaughtered Lamb – great now you can stand outside. The value in events like this is in the discussion.

See you there.

Jenkins talk in London, 31 May

Kohsuke Kawaguchi is doing a talk at Skills Matter, in London, on the 31st of May. You might know that he wrote Jenkins (nee Hudson). This should be a really informative talk, from someone who raised the game in Continuous Integration. I’m going to get there early to get a good seat. You need to register on the Skills Matter site, below. Get in quick.