Tag Archives: ant

All build tools began with Make

All build tools began with Make

(image taken from slashcrisis’ photostream)

Today I’m just going to share a pet hate: poor target names in build files. Yes, that’s scratching the surface: there’s plenty of other things to get wrong in your build. But today’s gripe is target names. Here’s an anonymised example from a real project:

<?xml version="1.0" ?>
<project name="project" default="tests" basedir="..">

<target name="run-database-tests" description="Runs all database tests"
depends="foo-run-database-tests, bar-run-database-tests, baz-run-database-tests" />

<target name="produce-docs" description="Produces javadocs for the project">
<echo message="Building docs for ${common.dir}"/>
<run-javadoc dest="${common.dir}" source="${common.dir}/src/java/com/company/project/common/"/>
</target>

<target name="produce-diagrams" description="Produces diagrams for the project">
<run-springviz dest="${foo.dir}" config="${foo.dir}/config/WEB-INF"/>
<run-springviz dest="${bar.dir}" config="$baradmin.dir}/config/WEB-INF"/>
<run-springviz dest="${baz.dir}" config="${baz.dir}/config/WEB-INF"/>
</target>

</project>

What on earth would you do but run the database tests and produce the javadoc? Aren’t the verbs (like run and produce) superfluous here?

What if instead of typing

ant produce-docs


You could type

ant db-tests docs

It all makes sense when you think that most build tools are evolved from make. If you replace the examples above with make instead of ant, it seems to work.

Make was written in 1977 by Stuart Feldman at Bell Labs. Just about every Unix distribution ever includes make (or a bastardized version, but that’s another story). Microsoft implemented it. If you’re interested in build tools, have a play with make: it’s part of their heritage.

For me once I fool myself into thinking that the name of build tool is a verb, you don’t feel the need to put a verb in every target.

Still, I doubt that Apache Ant would have flown with a name that was a tribute to make (Jake? Ache?). And what they have called NAnt? Don’t go renaming your build tools on my account; but when you write a target, pretend you’re building with Make.

Tagged

Oh lord, It’s hard to parse build files

Oh lord, It’s hard to parse build files

Nat Pryce left a comment on my post A real Build
Refactoring, in the wild:

IntelliJ can do some simple refactorings of Ant scripts: extract
property, rename target, rename property, etc.

But refactoring of Ant and Nant is very difficult because they have no
consistent syntax or semantics. They are just quick hacks that have
grown kludge by kludge into inconsistent pseudo-languages.

It seems I did Intellij a disservice when I said that it couldn’t do refactorings on build files. I downloaded the latest Mac release and had a try for about 20 minutes before I did the post. Refactoring didn’t happen. Maybe I should have spent longer. Sorry, Jetbrains. Thanks, Nat.

It does raise the question about our build tools. Maybe they’ll never have a decent editor. Nat’s multiple kludge theory would explain a lot. There’s a lot of editors that attempt to be your friendly build file editor, but not many come close. It’s been interesting watching Resharper 4′s Nant support evolve; it seems like it’s been hard work all the way.

For the time being, I’m reverting to the me of 6 years ago: I’m doing all my editing in Vim.

Tagged , ,

Ant Best Practices: Avoid platform specific wrapper scripts

Ant Best Practices: Avoid platform specific wrapper scripts(image taken from SideLong’s photostream)

Welcome to the final Ant Best Practices post. The last topic of the series is wrapper scripts. Most projects end up having one or more of these. The very first Java project that I did had about a dozen of them. They generally look like this:

#!/bin/sh
/usr/bin/ant -f myGreatProject.xml $1 $2 $3 $4 $5 $6

There’s a compelling reason for doing this: everyone appreciates being able to type three keystrokes to kick off the build. So why are Eric and I so against the idea? Because it invariably grows into a complex piece of code in it’s own right.

Ant already comes with a decent sized wrapper script to locate Java and invoke the ant classes with the right libraries:


jsimpson@knox:~$ wc -l `which ant`
317 /usr/bin/ant

The version that ships with Debian is 317 lines long. I think that’s enough wrapper.

It’s okay to have a one-liner shell script to call your build. Making it do more than that is the beginning of a very slippery slope. In the project mentioned earlier, the wrapper scripts (there were about a dozen different files, from memory) would locate a WebLogic installation, set the $CLASSPATH environment variable so that the application build could locate WebLogic. Eventually the wrapper would build the application and then exit, passing the calling process a return code of ’0′, regardless of success.

Once we upgraded WebLogic things got more difficult. Now there were 2 wrapper scripts that found different installs of WebLogic. And there had to be wrapper scripts for those, of course. Then there was the deployment scripts that:

  • Look up each developer’s assigned TCP port from a file
  • Start Weblogic and pass it the right port number
  • Deploy the application so the developers could poke around in the name of unix testing.

Luckily we had Dan North came in and replaced our entire polyglot build system with a single Ant script.

If you must write one of these then please consider the following points:

The parameters The job of your average wrapper script it to take parameters from the command line and pass them down to your build. In the example above, the script looks for six parameters and passes them. But what if you have seven? Scripts like this make me spit out my coffee. Deal with parameters like this:

  • In Bash shell scripts, use $@ to pass ALL arguments to the build.
  • In Windows batch files, use %* to do the same.

Always use relative paths, unless you absolutely have to. If your wrapper script has a relative path from your build tool (which should be checked in to the Version Control System like everything else), nobody has to mess with it. If you have a fixed path, it will start growing logic to accommodate everybody’s environment.

Summary: Be good. Don’t do them. If you can’t be good, be careful.

Tagged

Ant Best Practices: The Clean Test

Ant Best Practices: The Clean Test. Last time, we discussed the ZipFileSet. Today’s article is much more important – it’s a subject very close to my heart: only dong necessary work. If you don’t execute the clean target, (do tell me your build has a clean target and you didn’t change any of files in your project, what should happen?

I’ll tell you what should happen. Nothing. Diddly. Well, no compilation. If no compilation happened, should it bother packaging up artefacts or anything? I don’t think so. If you want to run tests, so be it; you might have an unreliable test. But for the most part, nothing should happen if nothing changed. This is the clean test.

Why? Think of it like this. If you change a single CSS file on your project, do you want to go and recompile all your code and run your tests? Hold that thought.

On the first and largest project that I ever did at ThoughtWorks, the fast build was 45 minutes. Don’t even go there with the build that did automated functional tests. One evening I was working late with one of the performance testers. The CruiseControl build was firing all the time. We realised that every time he did a performance test run, he checked in the results into the VCS alongside the code. That triggered a build. Whoops. We didn’t always have thatcapacity to spare. I have since sat there on many a project and watched builds trigger because a developer had to make the tiniest change. It’s dull to sit through a tedious CI build, just because the build didn’t discriminate between the files that had changed.

This stuff can make a difference. When your project has grown to a larger size, rest assured that it will be hard to change these things. So how do you convince Ant to be less zealous? Start by making sure that your default target doesn’t depend on your ‘clean’ target. When you refactor code and move classes around, you might need to execute the clean target. But treat that as the exception and not the rule.

That’s fine until you start packaging up the code in zip or jar files. What happens then? The uptodate task is your friend. This little guy will take a look at the sources and output of a task, and tell you if you should bother running it. Take a look:


<project default="archive">
<property name="build.directory" location="build" />
<property name="source.directory" location="src" />
<property name="archive" location="${build.directory}/stooges.zip" />
<fileset dir="${source.directory}" includes="*.xml" id="archive.files"/>
<target name="clean">
<delete>
<fileset dir="${build.directory}" includes="**/*"/>  </delete>
</target>
<target name="check-archive">
<echo message="Checking zip file time stamp" />
<uptodate property="-archive.is.unchanged" targetfile="${archive}">
<srcfiles refid="archive.files"/>
</uptodate>
</target>
<target name="archive" unless="-archive.is.unchanged" depends="check-archive">
<echo message="Making a zip file"/>
<zip file="${archive}" >
<fileset refid="archive.files" />
</zip>
</target>
</project>

The first time that you run this build, it does what you expect: makes a zipfile. The clever bit is what it does the second time you run it. It doesn’t bother. Here’s the first run:

Detected Java version: 1.6 in: /usr/lib/jvm/java-6-sun-1.6.0.06/jre
Detected OS: Linux
parsing buildfile /home/jsimpson/Documents/workspace/playpen/code/up2date.build.xml with URI = file:/home/jsimpson/Documents/workspace/playpen/code/up2date.build.xml
Project base dir set to: /home/jsimpson/Documents/workspace/playpen/code
[antlib:org.apache.tools.ant] Could not load definitions from resource org/apache/tools/ant/antlib.xml. It could not be found.
Build sequence for target(s) `archive' is [check-archive, archive]
Complete build sequence is [check-archive, archive, clean, ]

check-archive:
[echo] Checking zip file time stamp
[uptodate] The targetfile "/home/jsimpson/Documents/workspace/playpen/code/build/stooges.zip" does not exist.

archive:
[echo] Making a zip file
[zip] Building zip: /home/jsimpson/Documents/workspace/playpen/code/build/stooges.zip
[zip] adding entry iggy.xml
[zip] adding entry dave.xml

BUILD SUCCESSFUL
Total time: 1 second

You can tell that it was checking the lie of the land before it executed the main task. You might like the output from the second task:


Detected Java version: 1.6 in: /usr/lib/jvm/java-6-sun-1.6.0.06/jre
Detected OS: Linux
parsing buildfile /home/jsimpson/Documents/workspace/playpen/code/up2date.build.xml with URI = file:/home/jsimpson/Documents/workspace/playpen/code/up2date.build.xml
Project base dir set to: /home/jsimpson/Documents/workspace/playpen/code
[antlib:org.apache.tools.ant] Could not load definitions from resource org/apache/tools/ant/antlib.xml. It could not be found.
Build sequence for target(s) `archive' is [check-archive, archive]
Complete build sequence is [check-archive, archive, clean, ]

check-archive:
[echo] Checking zip file time stamp
[uptodate] iggy.xml omitted as /home/jsimpson/Documents/workspace/playpen/code/build/stooges.zip is up to date.
[uptodate] dave.xml omitted as /home/jsimpson/Documents/workspace/playpen/code/build/stooges.zip is up to date.
[uptodate] No sources found.
[uptodate] File "/home/jsimpson/Documents/workspace/playpen/code/build/stooges.zip" is up-to-date.

archive:
Skipped because property '-archive.is.unchanged' set.

BUILD SUCCESSFUL
Total time: 1 second

There you have it. It didn’t bother creating the zip file because the sources hadn’t changed. That’s useful. But what about the duplication?

Admittedly you do need to define the sources and target of the task twice. This could lead to duplication. If you look at the example, you’ll see how I got around that:

  • the target of the zip task is a property. So that’s fine.
  • the sources of the zip task are all types of FileSet. So I defined the FileSet once and passed the reference to both tasks.

Everyone’s build should undertake the Clean Test.

Tagged

Ant Best Practices: Use ZipFileSet

Ant Best Practices: Use ZipFileSet

(image taken from the superbly named MasochismTango’s photostream)


Welcome to the lucky thirteenth edition of Ant Best Practices. You probably guessed this one: Use the ZipFileSet type when you make a zip file in Ant.

This one slipped past me recently. We were working on a web project and the developers added a cache-busting feature to stop CSS stylesheets being cached by the reader’s browser. So they wrote build targets to:

  • fetch the static content
  • lay it out in a directory, nested under an arbitrary kind of key
  • zip it all up for deployment later

It looked something like this:

<project name="web" default="zipfile">
  <property name="build.dir" value="build" />
  <property name="tmp" value="${build.dir}/tmp" />
  <target name="zipfile">
    <copy todir="${tmp}/static/random_token">
      <fileset dir="code" />
    </copy>
<!-- more static files, you getthe idea -->
    <zip file="${build.dir}/static.zip">
      <fileset dir="${tmp}" />
    </zip>
  </target>
</project>


Sounds fine, right?

Hmm. Actually, no. The approach gets top marks for actually working, but where I should have intervened was the copying of the files about to make the paths that were desired. In this case, the <zipfileset> lets you pluck files from wherever they might be, and put them into the right place:

<project name="web" default="zipfile">
  <property name="build.dir" value="build" />
  <target name="zipfile">
  <zip file="${build.dir}/static.zip">
    <zipfileset prefix="random_token" dir="code"/>
  </zip>
  </target>
</project>


Really, that’s it. I was originally quite surprised that it made Eric’s original list of practices: It’s quite a simple change to make. But on writing about it I’m thinking this should be a refactoring. It’d be great to invoke an automated refactoring and introduce a zipfileset whenever you saw tedious copying and zipping operations. Anyway. Do this, and it will make your build faster and easy to read. Result.

Tagged

Ant Best Practices: Use Ant as the Least Common Denominator

We’re back to the best practices this weekend with 12 of 15: Use Ant as the Least Common Denominator. What are we talking about? The answer is here.

What does the common denominator mean? There’s generally a conflict around this on software projects. I’ll explain:

The developer wants to write code. Fair enough. That’s kind of their job. IDE’s give them an immense productivity boost in doing that job. If she needs to switch to another application, the developer will lose focus. So there’s a booming IDE plugin market. So developers will tend to avoid vendor tools in favour of IDE plugins: version control plugins, database plugins, tracking system plugins, etc. By doing so they can become very, very effective.

The release manager and his superiors have a different angle on this: they want to know that the code is deployable. To be deployable, you have to be able to build it. So they have Ant, or some other build tool. That way they know that the code will consistently build and pass unit tests.

Developers (like most people) don’t tend to like impediments to their productivity. Release managers aren’t fond of code that they can’t metamorphose into a working system whenever they feel like it. So developers would rather build all the code using an IDE than have to switch to a command line and build the code. Most release managers don’t want a dependency on a tool that doesn’t really address their needs. It can also be difficult to automate IDE builds.

So the conflict is that each camp has valid reasons for not liking the tools of the other. Which way do you tip the scales? Projects that aren’t constrained by developer productivity (i.e. maybe you have a shortage of testers) should be setting clear guidelines about the build tool. On the other hand, I’d get the kid gloves out if the developers were under pressure to deliver. Eric M. Burke suggests that you at least make sure that:

  • There’s an Ant build that the developers can use
  • They run it before checking in
  • They can use whatever tool they like while until checkin time.

That way, you know that you can reproduce the software later; that Bob didn’t check in code that builds against the development version of a library. The regular checkpoint of the pre-checkin Ant build will allow some flexibility for the developers.

It works for me.

Tagged ,

Ant Best Practices: Use version control

Amazingly, it’s article 11 of 15 in my series on Ant Best Practices. Today’s practice is ‘Use version control’ and I can’t help but wonder if this one hasn’t dated a little. When the original article was written, Subversion didn’t exist, Perforce wasn’t free, and most people used CVS. I’m not going to get nostalgic about CVS. It was better than the alternatives (ever used SCCS?). I wouldn’t go out of my way to use it now.

So would anybody actually challenge this anymore? The original article by Eric M Burke suggests that people would version code but not the build files. I’d be gobsmacked if anybody did that nowadays. Drop me a comment or note at ‘medic@build-doctor.com’ if you seriously disagree with version control for your build files. I’d really like to hear why.

I’m chapter and verse with Eric here:

  • Version control your code
  • Version control your build files
  • Version control your binary dependencies (unless you have an external dependency manager). Don’t version control build output.
  • And try not to mess around with checking things in and out of version control from your build. It generally gets ugly.

The sin of not using version control has done to death anyhow. Ask Joel.

Tagged

ant logfile /home/cruise/yourgreatproject/log.xml does not exist

ant logfile /home/cruise/yourgreatproject/log.xml does not exist

(image taken from losiek’s photostream)

Welcome to the first post of the Build Doctor that isn’t in the Planet TW feed. If you’re reading this via RSS: thank you. Thank you for subscribing. The point of this exercise was to actually know who my readers are. And now I know.

Back to the post. I’ve been helping someone set up Ant and CruiseControl. He mailed me last night to say that he got a log message saying that that the log.xml file didn’t exist. The short answer is that the Ant build failed.

The longer answer is that CruiseControl calls Ant with an argument to make it use the XmlLogger. Then CruiseControl can hoover up the XML file, and display the output in the Dashboard. What appears to happen is that if your build fails very quickly, the log file isn’t written. So all CruiseControl can do is complain about the missing log file. It all makes sense when you read the code:

if (!file.exists()) {                
throw new
CruiseControlException("ant logfile " + file.getAbsolutePath() + " does not exist.");
} else if (file.length() == 0) {
throw new CruiseControlException("ant logfile " + file.getAbsolutePath()
+ " is empty. Your build probably failed. Check your CruiseControl logs.");
}

Of course, there’s other reasons why this could fail, but every time I get this message, I find out that it’s down to a very broken build or broken Ant install. I’m tempted to try and reproduce this issue with the XmlLogger.

Tagged ,

Ant Best Practices: Keep the build process self-contained

Ant Best Practices: Keep the build process self-contained

(image taken from Heldr’s photostream)

Welcome back to the Ant Best Practices series. All the other posts are here, if you’ve just joined us.

Today, it’s about keeping the build process self contained. What does this mean?

When you check a project out and build it for the first time, it should take a few minutes at most to build it. I have done projects where it takes a long time to get started so you can actually build the thing. Funnily enough I’m in the process of tearing out a build system that required you to pass fully qualified paths down to it, so it can find bits of itself. When no developer checks out to the same place, you can’t consistently build it (when developers do check out to the same place, you get other issues).

This need to refer to the outside means that it’s really hard for anybody to build the project. When you introduce something like an application server that is outside the project, with environment variables to locate it, then things get even more complicated. The more that you can push into your version control system, the easier it gets. That’s why you find some developers checking enormous enterprise tools into the VCS; not because they think it’s the best place to keep things, but because you can always find them from your build.

Libraries are my favorite example. You can check them into you repository. You can’t diff them of course, but you can always find them. Some people really don’t like this behavior, and point out that VCS systems are for managing source files, and not binaries. They have a point. But what is the alternative? You could build everything from source (do you run all the tests as well?). Or you could try a dependency manager like Ivy or Maven. Maven’s dependency management is available as Ant tasks, by the way.

I’m looking forward to the release of Ant that comes with Ivy. It might make it feasible to manage binary dependancies. More on that in another post.

Tagged

NAnt vs Ant: locations (NAnt Rant)

NAnt vs Ant: locations (NAnt Rant)(Image taken from Nesster’s Photostream)

I think it was 2002 when Dan North took me under his wing and showed me the location attribute of Ant. That was then. Now, I’m doing a lot of .NET build engineering. And I’m dying for this feature. Here’s an Ant build to demonstrate:

<project default="properties">
  <target name="properties">
    <mkdir dir="build" />
    <property name="value" value="build" />
    <property name="location" location="build" />
    <echo>   here's the one with a value ${value}   here's the one with a location ${location}    </echo>
    <touch file="${value}/value.txt" />
    <touch file="${location}/location.txt" />
    </target>
</project>

Both the properties that are set represent a directory. Each has a relative path. What happens when you run it?

Buildfile: /Users/jsimpson/Documents/workspace/playpen/code/props-build.xml

properties:
[mkdir] Created dir: /Users/jsimpson/Documents/workspace/playpen/code/build
[echo]
[echo] here’s the one with a value build
[echo] here’s the one with a location /Users/jsimpson/Documents/workspace/playpen/code/build
[echo]
[touch] Creating /Users/jsimpson/Documents/workspace/playpen/code/build/value.txt
[touch] Creating /Users/jsimpson/Documents/workspace/playpen/code/build/location.txt

BUILD SUCCESSFUL
Total time: 1 second


Amazing. The property set with the location does the right thing and works out it’s fully qualified path. It also deals with any platform specific path seperator issues and presents you with the appropriate path. You may not think that this matters; the touch command worked for the property that used a value attribute, right? Yes, but only because that task will do the work. If you have a task or external command that doesn’t do the right thing, it’ll break.

Nant doesn’t have this. Here’s the same version, which works in the same way:

<project default="properties">
  <target name="properties">
    <mkdir dir="build" />
    <property name="value" value="build" />
    <echo>   here's the one with a value ${value}  </echo>
    <touch file="${value}/value.txt" />
  </target>
</project>

Giving us:

Buildfile: file:///Users/jsimpson/Documents/workspace/playpen/code/props.build
Target framework: Mono 2.0 Profile
Base Directory: /Users/jsimpson/Documents/workspace/playpen/code.
Target(s) specified: properties

Build sequence for target `properties’ is properties
Complete build sequence is properties

properties:

[echo]
[echo] here’s the one with a value build
[echo]
[touch] Touching file ‘/Users/jsimpson/Documents/workspace/playpen/code/build/value.txt’ with ’05/29/2008 21:04:25′.

BUILD SUCCEEDED

It gives the same result, but only because some tasks will work out that it’s a relative task and compensate. Other things won’t.
The astute reader might guess that I let myself get bitten by this again today. Maybe one day I’ll remember this, but right now I’d cheerfully leave the functions, and switch back to Ant in a heartbeat. Please, someone tell me that I’m wrong and that there’s a patch somewhere.

Tagged ,
Follow

Get every new post delivered to your Inbox.

Join 3,107 other followers