Outsourcing Continuous Integration

Outsourcing Continuous Integration isn’t a new idea, but we’re seeing more and more traction in the space. The headline news is:

  • It’s certainly not for everyone.
  • The space is going to get more and more interesting as cloud services increase.

Not for everyone

If you’re at all paranoid about security, you’re unlikely to want outsourced CI.

You might fall at the first hurdle: will you be able to justify outsourcing the build of your most valuable asset to an auditor? If you can’t address the (low) risk of your code being subverted, it might be game over. Perhaps you can prove that no code built at your outsourced service is used. It also raises the question of your version control system. Is it inside your firewall? Do you need to provide secured access to your outsourcing partner? Perhaps you outsource version control elsewhere. Can those parties talk? Could there be a man-in-the-middle attack?

What about some more practical reasons for keeping it in house? You might depend on internal services for your build. What’s your internet connection like? Do you mind if you lose your connection to the Internet, and therefore the outsourced continuous integration server?

Cooler tools

Can your IT department supply you with Linux, XP, Vista, and now Windows 7 with a host of different browsers? Of course not. They aren’t there to deliver a glittering array of choice in operating systems. Your friendly local IT department is there to drive down the cost of computing by stamping a uniform operating system onto all your computers. Your helpful IT vendor is there to help said IT department, being the guys who pay Bill’s bill.

It’s a good thing Amazon branched out from selling dead trees. The Amazon Web Services tool-set is amazing. Want somewhere to keep all those built artifacts? Then how about S3? Need a few dozen build agents? EC2 is your friend. We’re really just getting started here. One of the most obvious uses for the cloud is in allowing you to test all those pesky client configurations: those permutations of Windows, IE and Firefox, for example. I predict that Continuous Integration vendors will quickly reach feature parity on this, because it’s so darn useful.

Such services will become more specialised as more service models evolve. Need to test with your enterprise stack? I imagine you’ll be able to piece together some of those components as well. Will there be an API for submitting builds to any build farm? I certainly hope so.

In the medium term, I’m not convinced that many CI servers will end up fully hosted on the cloud. What’s more likely is that many enterprises will end up with:

  • One big, hand-rolled build machine, hosted at the firm.
  • Lots of nodes in the cloud.
  • A really freaking big Amazon EC2 bill.

This works, because you get to assume that you’re protecting the your assets, and just giving your built code a workout out there in cloud-land (I also predict the rise of compromised cloud servers, FWIW). You still need to deploy the app somewhere and fire up nodes to test against it, but you are limiting the opportunities to inject malicious code at build time. This allows you to keep built artifacts (be they in a Maven-style repo, or just spat out from an Ant build) on the inside of your network (ironically where you probably face the most realistic risks of attack – by disgruntled or financially compromised employees).

Perhaps some of the cloud vendors will acquire enough security certifications to convince auditors that it’s safe to use. And maybe, enough organisations will start thinking of operating systems and middleware as bigger code objects to play with via an API or toolset, rather than infrastructure to manage with a meatcloud.

Some vendors

So who actually provides outsourced Continuous Integration? This is by no means an exhaustive list. Tweet me if you have suggestions for the list. Thanks.

  • Collabnet offer Team Forge, which looks like it used to be SourceForge Enterprise Edition. Remember that? I worked at a bank that used it. Happy times. [mainly due to NPR and Peet’s Coffee. Though SFEE did work reasonably well for a large programme of work]
  • Run Code Run – have built off the back of GitHub with a sweet little model – they consume hooks from GitHub, and trigger from those to build your Java apps. They are branching out from Ruby projects to include Java as well, and will rent you a private CI system by the month.
  • CI in a Box is an Amazon EC2-based solution. I’m not sure who’s making money off of this one apart from Amazon – the house always wins. Looks like low cost and scalable Hudson implementation, anyway.
  • Mike CI contacted me the other day – they have a new service – operated out of the UK, but available everywhere, of course. They are pre-launch, but they seem to be in a similar space to Run Code Run – allowing developers to easily adopt CI. They support Java but might also offer .NET. I’ll try and get something more in-depth, and pounce on them for an interview if they come to London. They seem really nice.
  • Atlassian just joined the game with JIRA Studio, their outsourced suite of tools. This is a good play from them: they have a strong brand in JIRA, and they are leveraging it.
  • Electric Cloud offer a tool that can be fully or partly cloud hosted.  It’s not clear who offers this as a managed service or not.  I’ll ask them.
  • Bitbar are new.  Looks like they have a strong mobile vertical.
  • Hosted CI got in touch as well. They self-describe as “Hosted Continuous Integration for iOS and Mac”
  • TDDium (geddit?) said hello recently. They are “a cloud-based test environment designed to change the way developers build web applications”. Or as we call it, Continuous Integration.
  • CI Foundry is also new, and in super-alpha. This is a bespoke service, so aimed at companies who want things done for them, or in situations where the standardised offerings don’t fit. DISCLAIMER: I’m behind this one. I’m going to be open about this. Compromising my editorial integrity would feel dirty. I’ll even try and get someone else to do reviews if there’s a problem.

Are you using outsourced Continuous Integration? Do you want to share your experiences? Tweet me!


Added Atlassian on December 17, 2009

Added Bitbar on June 14, 2010

Added hosted-ci, and removed Run Code Run and Mike, September 30, 2011

Added TDDium on December 29, 2011


12 thoughts on “Outsourcing Continuous Integration

  1. Very interesting stuff! Nice to see longer post from you.

    Sometimes it happens that in a busy space like this, someone hits on the magic combination of features and cost and service and takes over all of a sudden. Google and iPod are easy examples. Wonder if cracking the security conundrum is what will do it here. It is certainly one of the very few barriers we have to adopting cloud CI.

  2. Another thought: you are right that all vendors may soon offer a wide variety of modern OSes on their build agents. A way to differentiate would be to offer _old_ OSes. Imagine the pain of a company who has important clients still on Windows 2000 or 98 (this is not unheard of, believe me). You can’t even buy those systems any more except maybe on ebay or something, and forget maintaining them (even in a VM they must be a nightmare). Dev teams in this situation may have no other way to run CI other than to outsource to someone who can afford the setup cost.

  3. […] Published October 9, 2009 Uncategorized Leave a Comment We got our first mention in the blog-o-sphere today with a good article on the pros and cons of outsourced continuous […]

  4. simpsonjulian says:

    @douglassquirrel, thanks for the insightful comments. I’ve pondered the security a lot. I guess the problem is going to be cost for security. Nail that one and you’re done. I think VPN’s are going to resolve a big chunk of those issues, and proving that there’s no theoretical risks to be had at the other end. For example, I could outsource someone’s CI services to a dedicated machine in a secure enclosure in a datacenter, It wouldn’t be cheap, but I could do it. Finding clever ways to resolve that problem would be the challenge.

    Winning some Ebay auctions to test out ancient operating systems would be a laugh. The cost would be securing Windows 2000 and 98 from the outside world, probably with an old-fashioned Cisco firewall to block traffic from anywhere but the CI systems. You’re right in that someone would have to scale up to make it worth while. Or the other approach is to try and run them all in emulators, or Wine, but that might invalidate some testing.

  5. No, no! Cost is _not_ the first issue for security of cloud CI. Steps have to be:
    1. convince techie that cloud CI is a good idea. (Existing providers do this.)
    2. convince techie that cloud CI will pass audit. (Need reference customer. Chicken and egg.)
    3. convince auditor to pass cloud CI. (Requires pen and the ability to tick boxes all day. Your first customer will give you a sample list – how did you get that first customer again?)
    Only consider cost once you’ve solved 2. and 3. The costs of in-house CI are so high that once you’ve done this you can almost certainly find someone willing to cover your bills plus a decent margin.

  6. @douglassquirrel: Okay. I guess that the elasticity of price depends on the size of organisation and the domain. Certainly what you just said is about 100% true for companies that do financial services.

  7. […] One area which is less mature is hosted continuous integration. There are currently only a small number of pioneering providers in this space, which may surprise some, as the practice of continuous integration is at the heart of the Agile development process. The SaaS multi-tenant application model does not fit easily with the requirements for continuous and often complex software builds. It is computing resource intensive activity, especially for programming languages such as Java, and this will inevitably impact the cost of such a service to the end-user. Mike CI is one of these pioneers and there is a good analysis of the others here. […]

  8. My take on this is that hosted CI in a common virtualized environment such as EC2 won’t work. A CI or a build server, unlike the rest of the applications, needs all four components of a build box, CPU, RAM, disk and network I/O. The industry wisdom says applications that are subject of virtualization may demand maximum two. Sure you can run a build in EC2, but you will have to sacrifice build speed, and that’s usually the last thing you want to do. If you want fast builds, you have to run in the opposite direction, towards a dedicated, big fat box hosted locally.

    Viewtier has been hosting Continuous Integration for open source projects for five years, and our experiences shows that even builds on a dedicated build box begin to slow down if the number of long-running builds exceeds a double of number of CPUs. Actually, we observe a trend towards farms of build machines hosted locally.

  9. […] batter your database for a long time while staying almost idle. Slava Imeshev kindly commented on myoutsourcing continuous integration post: My take on this is that hosted CI in a common virtualized environment such as EC2 won’t […]

  10. Disclaimer: I'm working for company that provides CI on Cloud to enterprises.Having some experience in the subject, I'd like to remove two myths :)Slava, you are right that virtualized EC2 environment is not good for CI. However for example Bitbar provides pretty fast CI on the Cloud – simply the machines are not virtualized! And you can easily add more machines if you have many long-running builds.A comment on the security of outsourcing CI.You can get exclusive VPN access to the cloud, so that even your CI provider wouldn't be able to access your data on the Cloud. Moreover, the machines holding the data wouldn't be able to access or get accessed from the internet in any way. I think that this level of security is more than enough for most of the 'paranoids' mentioned in the article.

  11. Hello again! One of my customers has to comply with not only his own regulatory environment, but those of his customers. They'll regularly send auditors around to make sure that there's no realistic potential for fraud in the development process. So, yes they may be paranoid, but it's probably not a paranoia that can easily be treated. VPN access may help some.

  12. At AppVeyor CI our build servers are Medium VM instances on Windows Azure and they are really fast! 😉

Comments are closed.

%d bloggers like this: