Category Archives: Java

Repository? That’s not a repository …

this is a repository. I’ve been experimenting with Amazon S3 as a Maven repository. I threw the results up on GitHub [link].

Why? I’ve been working with some nice people who use binary dependencies. In order to scale their CI system past one node, they need a repository manager, to temporarily store built artifacts on.

They actually have a couple of repositories, but not geographically close to where I’m building out their new CI system (on EC2). It’s important to keep the feedback loop fast; so deploying a repository close to(or indeed on) on the CI server is desirable.

The plan was to deploy Archiva onto an EC2 instance via Puppet. Then I had a better idea: use an S3 bucket as a highly scalable, low latency and (mostly) highly reliable repository.

Turns out that there’s support for S3 in their build tool (Maven). So it turned out to be easy. In the github repo, I publish and artifact using an S3 client, and then retrieve it using plain HTTP.

As long as your EC2-hosted Continuous Integration server is in the same availability zone as the S3 bucket, you’re not going to be liable for a high traffic bill.

They may want to use a different configuration for the real repository, where they serve artifacts to the other consumers. This approach is all about speeding up and scaling out their build.

Update: Oliver Lamy asks: ‘what is wrong with Archiva’?

Nothing. I’d cheerfully use it. It has a permissive license and cheerful absence of public spats with other repo managers.

In this case I wanted some very fast and robust, which led me to S3. The alternative was to deploy Archiva on an EC2 instance: I could have done that, but then I would have had a dependency on a single host.

IMO, this approach is more suitable for the context (caching intermediate build artifacts, on AWS).


Tagged ,

Is Maven agile?

Is Maven agile?Is Maven agile? Here’s my half-assed reasons why:

  • agile methodologies can uncover issues in your organisation. I’ve got a pet theory that Maven will do the same for your code.  Some build tools allow you to paper over the cracks of your code and tests.  Tests too slow?  Then parallelize them.  Code won’t compile cleanly?  That’s okay, you can frig the build file until it does.  Maven doesn’t help you do this kind of thing.  Maven makes you do the right thing.  I’m sure there’s a plugin that spits at your feet for even thinking about accommodating software quality problems in your build tool.
  • agile engineering techniques ensure that we can continue to deliver at an appropriate pace.  I was arguing about Maven with Steve Freeman recently; he pointed out that an Ant build should be a trivial thing to write.  What I didn’t say is that once you’ve got your maven build sorted, the plugins that you can employ come at a very low cost: Cargo and Sonar can save you literally hours of fiddling with XML.  So don’t compare time spent implementing the two tools so you can compile your code; look at what you have to do over the whole lifecycle.

Disclaimer: Your Milage May Vary.  Maven may not suit your needs.  Ditto with Ant. The value of your technical debt may go up as well as down.

(Image via http://www.flickr.com/photos/zenera/)

Ant will eat itself

I was sorely tempted to save this for April Fool’s Day.

<project default="default">
  <target name="default">
    <delete file="${ant.file}"/>
  </target>
</project>

Update: Yes it does work:

Socks:tmp jsimpson$ mkdir demo
Socks:tmp jsimpson$ cd demo/
Socks:demo jsimpson$ vim build.xml
Socks:demo jsimpson$ ls -l build.xml
-rw-r--r--  1 jsimpson  wheel  119  1 Feb 21:13 build.xml
Socks:demo jsimpson$ ant
Buildfile: build.xml

default:
   [delete] Deleting: /private/tmp/demo/build.xml

BUILD SUCCESSFUL
Total time: 0 seconds
Socks:demo jsimpson$ ls -l build.xml
ls: build.xml: No such file or directory
Socks:demo jsimpson$ 

dbdeploy.net

dbdeploy.netAgile Database deployment for Java and .NET

(This post was originally hosted at http://www.dbdeploy.net)

DbDeploy is an implementation of the ActiveRecord Migrations pattern. DbDeploy.NET is the .NET port of DebDeploy. Both DbDeploys are projects initiated by ThoughtWorks. ActiveRecord comes to us via DHH.

Why would I use it?

When you’re developing software that hasn’t been released, the database is easy: you can tear it down and rebuild it at will. Once you have production data that people are using it, what do you do? How do you manage the change? The Migrations pattern allows you to make bite-sized changes to your database, and test it. It works very well with Continuous Integration.

What else is out there?

Payware:

Open source:

When should I use this pattern?

It’s ideal for greenfield agile projects where you are using Continuous Integration and want to make sure that changes to the database schema will be applied to integration tests. You can use other approaches if you have an ORM and you haven’t released to production yet.

When shouldn’t I use this pattern?

  • When you have a huge legacy database
  • When you’re trying to put data into a database and not schema changes
  • When you don’t use source control

The Migrations pattern is a really helpful way to manage database change; It’s not a silver bullet though. You need to have discipline and a good test regime. It works well with Continuous Integration.

Update: Gregg Jensen got in touch with a new URL for DbDeploy.Net

Bamboo 2.5 released with improved Maven support

The Atlassian guys just released Bamboo 2.5.  Highlights are:

  • Maven Dependency Management
  • Plan Import from a pom.xml
  • Additional Bulk Actions
  • Streamlined Plan Creation
  • Express Setup Wizard
  • Plus over 70 fixes and improvements

Being able to manage dependencies in your CI tool is key if you’ve drunk the dependency manager Kool-Aid.  That way you get to flush out integration issues.

Bamboo 2.5 Release Notes – Bamboo 2.5 – Atlassian Documentation – Confluence.

(image from delphaber)

Ant Contrib: the power and the pain

I got a comment from Errno on my last post:

“Everyone agrees that Ant Contrib means you’ve done something wrong.” why? is there any blog post/write-up describing this? thanks

I don’t know of any write-up. So I’ll make one. There’s a reason why some tasks are in ant-contrib and not the Ant tool proper: they don’t really fit the tool. The culprits that I have in mind are trycatch, for, and variable.

Now don’t get me wrong: these keywords would all be great in dozens of languages. Catching exceptions: Great! Iterating! Sweet! With a local variable! Sweeter!

But not in a declarative language. Because that’s what Ant is. It’s a declarative DSL for building Java code. Declarative programming allows you to present the facts to the language, and let it go do all the plumbing. So flow control in your Ant scripts aren’t needed, if you go about it the right way.

What’s more, properties are immutable. You need never test to see if a property is set before setting it. They did that for a reason: it’s subtly very powerful.

Perhaps it’s too subtle. Ant’s a powerful tool if you go with the grain. There’s a couple of other things to consider if you find yourself using the Ant-contrib stick:

  • Should you be using Ant at all? Remember, it’s a build tool. If you’re bending the tool to fit your needs, perhaps another will be better.
  • If you do need to get a bit more imperative, perhaps you could write an Ant task and share it with the world? Then we all get to enjoy a simple build.

Yes, I’m a build tool socialist. Want examples? Send me a snippet of build and I’ll tell you how I’d do it The Ant Way.


Image via merfam

Tagged

Better Apache Ant Builds

Abstract: We did two CITCON sessions here; Joe Schmetzer suggested a show-and-tell about tools to reduce Ant bloat, and Ivan Moore wanted a general session on improving the quality of Java builds in general. During the conference, the two sessions merged. My plan was to sit in and tweet smart-ass comments. As I had my laptop open, they made me official scribe. Serves me right. Here’s my notes:

Joe: I have template Ant stuff to show you.
Ivan: I don’t like Ant. Is it just me?
(someone mentions how annoying it is when you have re-entrant AntCalls)
Chris Read and Julian Simpson: “Antcall is evil! Don’t use it!”
Ivan expands on his frustration with Ant: he builds something in Ant and then regrets it later. It has all the building blocks for the task at hand but he still finds it unwieldy.
Chris suggests orchestrating calls to Ant files using a scripting language.
Jeffrey Frederick suggests that writing custom Ant tasks is a good approach.

We discuss alternatives to Ant. None are found. Maven, Buildr and Gradle are mentioned but the broad agreement is that there’s always a trade-of between the convenience of those tools and the flexibility of Ant. [Please direct flames to the comments of this post. I really want to discuss this more]

JTF comments that the junior guy always gets to write the build. Why does the build get treated like a second class citizen? JTF suggests PowerPoint called object oriented build scripts, and a tool called EasyAnt.

Pavel from JetBrains mentions that IntelliJ have switched to Gant, and are very happy with it. I don’t believe that the all of the JetBrains Java projects have moved over.

Joe demonstrates his templates on a 9-inch netbook. This doesn’t really work out as we can’t get the projector going, so he outlines it. This is hard to transcribe.

Joe: Don’t cut and paste Ant scripts. Import predefined targets instead. Well defined targets can be reused. This leads to tiny Ant scripts for each project. Joe’s projects end up having a 10-15 lines of Ant script. Joe has released this all at Ant Script Library.

Joe explains the targets that he uses most frequently – the usual suspects of compile, test, etc. He’s integrated a lot of static analysis tools into the library (he rattled off a huge list, which I failed to capture) – that’s a win.

Joe‘s abstracted SCM and release targets nicely, by the sound of it.
Paul Duvall asks if they are targets or macrodefs. They are targets.
Joe hates code generation so doesn’t use it in this toolset. Healthy discussion of script generation.
There’s clear parallels with maven and the strongly defined code tree.
You can buy my refactoring ant article at at good second hand bookstores.[link]
People go against the grain of the tools.
Everyone agrees that Ant Contrib means you’ve done something wrong.
. JTF suggests that string literals are okay in builds.

How do you test build stuff? JTF suggests:

  • a test project
  • writing junit tests

Pavel and friends at JetBrains use personal builds to test all this.
Andy Parker asks how much container stuff and deployment you should use.
We all agree that running the production deploy in CI is the business.
Ivan: how do you deal with the devs on windows/prod on unix business
VM’s are the saviour of the windows/unix divide.
The windows/unix divide is a perennial issue.
Jetbrains use build agents to try and flush out issues.

JTF: Ant builds can suck when:

  • junior people write them
  • there’s no refactoring
  • they are treated as a second class citizen
  • imperative vs declarative
  • code ownership: if you touch it, you own it. Hence, nobody will touch it.

We had a vigourous discussion of dependency management. Chris asks for discipline. Andy points out that you can end up with integration hell. Maven’s release plugin can allegedly resolve dependencies too late in the QA process and end up changing the release candidate. Chris complains that it can be hard to rebuild later when you depend on the latest version of a dependency. Joe did his own wrote-up here.

PS: I’ll add some links back to the various people mentioned here

Tagged ,

Apache Ant script reuse

The Maven guys have got one thing (and others, I’m not being mean) right – we reinvent the wheel too much on Java projects.

Joe Schmetzer has an Ant Script Library. And Pascal here has had a crack at the problem of Ant Script reuse.

Seems a bit 2002-esque to write an Ant build from scratch these days.

Update: this tweet from JTF suggests another tool.


Tagged

CruiseControl Build Radiator

Sudhindra Rao from ThoughtWorks has released a Build Radiator (Big Visible thing) for all the CruiseControls that support CCTray.

I installed it tonight. Needed to install the rack and activerecord gems. It would be nice to see it wrap around if you have more than a few projects. Most teams would probably do fine with this on a big display, I suspect.

This is pretty much what Michael and I did last year, except Sudhindra has made his nicely generic. Oh, and open source and available to the public. That helps a lot.

Nice work, Sudhindra.

Link


Tagged

Build Refactoring: Delete Boolean Property

(image taken from Kurtis Scaletta’s photostream)

Last time, you read about the extract target refactoring. Today, I’m going to show you one that I love to perform: delete boolean property. Have a look the smell first:

<project default="unit_tests">

	<property name="junit.enabled" value="true" />

	<target name="-check_junit_enabled">
		<condition property="junit.really.enabled">
      <and>
        <equals arg1="${junit.enabled}" arg2="true" />
      </and>
    </condition>
	</target>

	<target name="unit_tests" if="junit.really.enabled" depends="-check_junit_enabled">
		<mkdir dir="build/test-results"/>
		<junit>
			<test name="FooBarImplTest" todir="build/test-results"/>
			<classpath>
				<fileset dir="build"/>
				<fileset dir="lib"/>
			</classpath>
			<formatter type="xml"/>
		</junit>
	</target>
</project>

This smell starts out with the notion of properties to control the execution of your build. So far so good. “Hang on”, someone pipes up: “If you’re going to have a property, I want to be able to set it to values like ‘true’ or ‘false'”. With more work you manage to express the logic in Ant. Each target now needs to depend on at least one other to set the properties that control the execution of the build. This rapidly turns the build into dependency soup.

You also have to set two properties: the one that has the boolean value, and the other that exists to actually signal to your build tool that it should run a target.

How do you resolve this? Start deleting all those extra targets that ensure that the right properties are set. Like this:

<project default="unit_tests">

	<target name="unit_tests" unless="junit.disabled">
		<mkdir dir="build/test-results"/>
		<junit>
			<test name="FooBarImplTest" todir="build/test-results"/>
			<classpath>
				<fileset dir="build"/>
				<fileset dir="lib"/>
			</classpath>
			<formatter type="xml"/>
		</junit>
	</target>
</project>

What do you have to change in the targets that do the work? Just choose a decent property name to put in the if or unless tags and then delete any dependencies on targets that resolve properties. I decided that the default behaviour should be running the unit tests. To opt out you can pass the argument -D:junit.disabled=foo on the command line.

So what about the person who wanted the boolean properties? Either:

  • Make them support the build,
  • Write a custom Ant task to do all this, and make it run in the body of the Ant build file and not inside a target. I’ll even write one for them if they dig deep enough.

Happy refactoring. If you like this post, my article on Ant Refactoring is available to buy in the ThoughtWorks Anthology:


Tagged ,