Category Archives: Systems Administration

Zabbix, narrated by Jon Topper

The London DevOps crew recently met up at The Guardian’s fabulous King’s Cross building to eat, drink and talk devops. The first talk of the evening was Jon Topper, sharing his experience with the Zabbix monitoring platform. I plan to investigate as I’d rather eat catfood than configure Nagios again.

I’ll post the next talk on ‘Git for sysadmins’ in the next couple of days. This seemed to be the biggest formal meeting yet. Quite pleased about that.

Thanks to:

Tagged , ,

DevOps is a good cause, but what about OpsOps?

A few recent blog posts have attempted to explain the Devops bloody revolution movement. I’m overdue to post one of my own. Lest we get carried away, let’s not just focus on the Developer Systems Administrator axis. As Graham expounded on Friday night (and I paraphrase, for the Build Doctor had prescribed himself ale):

We need Ops-Ops! There’s so many teams where the admins hate the DBA’s, and the Networks teams hate the admins. And the DBAs aren’t fond of anyone. There’s enough trouble getting those guys to work together.

He’s got a point. Those damn siloes are everywhere.


London DevOps Meetup

There’s some fantastic DevOps talent in London.  It’s been a little fragmented, mind you. That stops on February 24th.

Come to the pub and talk about delivering great software without status, remit or pissing matches.   If you can’t make it, there’s always the #ldndevops Twitter hashtag, and the London Devops blog aggregator (see the link below).

London DevOps.

Giternal Debian package

Giternal Debian packageGiternal is a handy tool for managing git submodules. Give it a YAML file of git repositories, and it’ll ensure that the repos are all checked out. I’d use it to bootstrap all of the Puppet modules that I need, except for one fact:

It’s a rubygem. Happy to use them, but I don’t want dependencies on Rubygems at the bottom of my toolchain. I also have a boostrapping problem because Rubygems get installed from a Puppet module, which I fetch using Giternal.

What to do? Make a Debian package! So I did. You can download it here. It depends on ruby. I can make a Debian repository if anybody likes that idea.

Here’s a tiny example of the yaml file.

  path: modules
  path: modules

Here’s the giternal Debian package install.

jsimpson@fox:~/Documents/workspace/giternal$ sudo dpkg -i giternal.deb
(Reading database ... 145351 files and directories currently installed.)
Preparing to replace giternal 0.1.0-debian1 (using giternal.deb) ...
Unpacking replacement giternal ...
Setting up giternal (0.1.0-debian1) ...

Here’s giternal doing its thing.

Socks:puppet jsimpson$ giternal update
Updating common
Updating wordpress
Updating sudo

Image via (Git book author) tswicegood


Rubygems on Ubuntu (with Puppet if you like)

Rubygems on Ubuntu (with Puppet if you like)

Debian packages and Rubygems: they get on like two angry cats in a sack. This post explains how you get Rubygems and Dpkg to play nicely on Ubuntu Hardy Heron.

What’s the issue? The Debian Packaging System (DPKG) is pretty good as packaging systems go. It’s had dependency support baked in for years, which means you can install one package, and any package that it needs, and so on. Rubygems is good in other ways. It’s a very convenient way to use and distribute other people’s Ruby code. Rubygems is optimised for developer convenience. The Debian system is optimised for stability. What happens when you try and make them work together? Bloodshed.

How do we cope? Most people get around it by installing the stock Rubygems into the Debian or Ubuntu system. This overwrites ‘/usr/bin/gem’ with a version of Rubygems that will install gems into the ‘/usr/’ directory on your system. What happens if you upgrade your Ubuntu distribution? The new version of Rubygems will overwrite the stock one. Because the Debian maintainers changed rubygems to install gems into a safer place (/var/lib/gems), you’ll probably notice installed gems disappear.

But what’s the downside? When the new Ubuntu LTS (Lucid Lynx) is released in April, you’ll probably want to upgrade. I’m finding Hardy harder to support as new tools arrive (couchdb is a good example). Is it time to get your systems sorted out now, so that you don’t break your apps on the upgrade?

How to do this? You can install the rubygems packages from Karmic Koala on Hardy Heron. Have a look at these instructions below. Note: this might make your Gems disappear. Do have some gem list –local output so you can put them back!

# HINT: run the sudo first or it will swallow any other text
sudo -s 
# rubygems and this scripts's dependencies
# if they are already installed, it won't reinstall
aptitude install wget ruby1.8 rdoc1.8  

# get the new gems on the system
cd /tmp
dpkg -i *.deb

# tell your system about them
cat <<HERE > /etc/profile.d/
  export GEM_HOME=/var/lib/gems/1.8
  export GEM_PATH=/var/lib/gems/1.8

You’ll need to log out and back in again to see the changes. I also have this handy Puppet manifest to do all this en masse. Comment if you want me to publish.

Image thanks to Jijy

Update: I also want to help try and get rubygems working out of the box in either Debian or Ubuntu. Watch this space. Hopefully I’ll look at this over Xmas.

Update: Thanks to @auxesis for the comment. I’ve made a couple of changes to this post.


I’m at ScaleCamp today. Will hopefully get some blog posts done today, and will be tweeting.


Drunken Build Monkeys, and Agile Systems Administration

LRUG kindly let me do a talk on Systems Administration on Wednesday night. It had a working title of Drunken Build Monkey (my homage to Jackie Chan), but ended up with a far more serious title.

John Arundel opened the event (and let me steal his MacBook for my Keynote presentation). Thanks to the DevOpsDays crew for coming to the talk and joining in the conversation.  Here’s the video:


And here’s the slides:

Dtruss: Strace for OSX

(I just found this in my drafts, from 2009)

If you’re an old-school Solaris admin like me, you’re be used to running the truss command to see what applications are doing under the hood. Truss shows you what system calls a process is making. This is incredibly useful when you’re trying to diagnose library loading issues or timeouts, because you can see what the errant process is attempting to do.

Linux has strace, and Mac OS X inherited ktrace from the BSD family. They were all different, but incredibly useful to the troubleshooter.

Starting with Solaris 10, Sun upset the applecart by introducing dtrace. Dtrace isn’t a tool, but a complete language. Apple furthered this by nicking it (well, it is open source) and putting it in Mac OS X, while yanking ktruss.

Anyway, if you find yourself on a system that doesn’t have ktrace, don’t panic. This guy wrote an implementation of truss in Dtrace, and it’s part of Darwin:

knox:~ jsimpson$ sudo dtruss ant
SYSCALL(args) = return
madvise(0x7FFF8AD8E000, 0x2000, 0x5) = 0 0
open("/dev/dtracehelper", 0x2, 0x7FFF5FC45348) = 3 0
ioctl(0x3, 0x80086804, 0x7FFF5FBFD460) = 0 0
close(0x3) = 0 0
stat64("/usr/lib/libstdc++.6.dylib", 0x7FFF5FBFC870, 0x7FFF5FBFCEB0) = 0 0
read(0x3, "Darwin\n", 0x80) = 7 0
read(0x3, "", 0x80) = 0 0
wait4(0xFFFFFFFF, 0x7FFF5FBFE32C, 0x1) = 90163 0

All good.

A solution to broken Gems?

RubyGems have been exceptionally successful as a way for Ruby developers to share code. We generally think that sharing code is a good thing. Certainly, the Rails community can write projects exceptionally quickly; thanks, in part to RubyGems.

My beef has been that RubyGems doesn’t play nice with operating systems. We have standards for where things should live in any operating system. In the world I inhabit most (Unix systems), there’s a well defined place for everything. This is more than just being tidy. It helps keep things consistent and stable: when you install new software, dependencies can be easily found, for example.

Gems don’t really work like that. You can have many gem installations on your system, and they can all provide code to an application. It’s for this reason that the Debian team took the RubyGems code, modified it, and put it in a Debian package. This helped bridge the worlds of Debian packages (a packaging system that does dependency resolution to work out which packages you need in order to run the packages you want) and RubyGems (which mostly does the same thing).

Unfortunately this often lags behind the rapid development of RubyGems, so we’re forced to do things like install the newer version by hand over the top of the Debian version. This gets the job done, in doing so breaks the packaging system, which is a key reason that people use Debian or Ubuntu. Think it’s just me complaining about this? Some other people who know a lot about operating systems aren’t happy either:

At least for Debian or Ubuntu users, there’s one solution: DebGem. DebGem (from the guys who brought you Phusion Passenger and Ruby Enterprise Edition) bundles RubyGems inside Debian packages. If each package has its dependencies declared right, you’ll be able to install any gem and it’s native dependencies, in one go.

I hope it works.

The issue here is that it takes lots of time to maintain packages. Especially when you’re trying to track the packages that come from an extremely prolific community that knows how to use distributed version control systems (git is now the VCS du jour for Rubyists). Let’s hope that Phusion can charge enough to make it worth their while to continue; once you’re hooked on this, it would be hard to stop.

Not sure that this really solves the problem. The core issue here is that the developers are elegantly solving the problem of how to reuse code as developers. Systems administrators are trying to make systems that are stable and easy to maintain. The two might never meet.

Final thoughts: they might do well pursuing a freelance model to get gems packaged. Also, a commercial distribution might buy it to gain market share. My fee for this idea is a snip at 10%.

( image from Ed Yourdon )

Guest Post: Test First in Operations

Today’s post is by Matthias Marschall of the fantastic Agile Web Operations blog. Matthias is the CTO of Autoplenum, when he’s not blogging.

Monitoring a server is very similar to continuously building and testing code.

In development, you write tested code, commit it to version control and the CI server tries to build and deploy the code to a test system and executes all tests.

In operations, you write a script (e.g. a puppet manifest or a capistrano recipe) describing the desired setup of your server and commit it to version control. A daemon like puppetd deploys the configuration changes to a test system. But who takes care of testing your new setup?

Testing Configuration Changes with Monitoring Tools

Just like development, it’s a good idea to make sure that there’s a way to verify every critical aspect of your server’s configuration. In operations, monitoring services like Nagios act as tests. They are the last step of your operations build chain: Verifying that your deployed configuration changes work as expected and have no undesired side effects.

Test First In Operations

In operations, the fundamental guiding principle is often never touch a running system.

In development, you tend to think the same only about untested legacy code. Having tests, you can touch your code anytime as you know that your CI system will catch any mistakes.

A test driven approach is possible in operations as well, but I’ve hardly seen it anywhere:

  1. Write a monitoring service for the config change you want to make and ensure that monitoring goes red.
  2. Implement the config change on the system and watch your monitoring system go green.
  3. Rinse and repeat until you’ve implemented all desired configuration changes.
  4. Like in development, you should have a test server for deploying and testing your configuration changes before deploying them to production. No shortcuts here.

Having tests makes you fast and safe, no matter whether you’re developing your web application or operating it.

Image by Docklandsboy