I want to share something a software developer or operations guy every now and then encounters. And which I did yesterday.

It all started with our nightly Maven build failing last Saturday Jan 28 on a

java.lang.NoClassDefFoundError: javax/xml/bind/ValidationEventLocator

The interesting snippet from the concole output from this build looked like this:

[INFO] <b>[jaxb2:generate {execution: xxx-schema-gen}] </b>[FATAL ERROR] org.jvnet.mjiip.v_2.XJC2Mojo#execute() caused a
linkage error (java.lang.NoClassDefFoundError) and may be out-of-date. Check the realms:
[FATAL ERROR] Plugin realm = app0.child-container[org.jvnet.jaxb2.maven2:maven-jaxb2-plugin]
urls[0] = file:/home/hudson/.m2/repository/org/jvnet/jaxb2/maven2/maven-jaxb2-plugin/0.8.1/maven-jaxb2-plugin-0.8.1.jar
urls[1] = file:/home/hudson/.m2/repository/org/jvnet/jaxb2/maven2/maven-jaxb2-plugin-core/0.8.1/maven-jaxb2-plugin-core-0.8.1.jar
urls[2] = file:/home/hudson/.m2/repository/com/sun/org/apache/xml/internal/resolver/20050927/resolver-20050927.jar
urls[3] = file:/home/hudson/.m2/repository/org/sonatype/plexus/plexus-build-api/0.0.7/plexus-build-api-0.0.7.jar
urls[4] = file:/home/hudson/.m2/repository/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.jar
urls[5] = file:/home/hudson/.m2/repository/org/jfrog/maven/annomojo/maven-plugin-anno/1.3.1/maven-plugin-anno-1.3.1.jar
urls[6] = file:/home/hudson/.m2/repository/org/jvnet/jaxb2/maven2/maven-jaxb22-plugin/0.8.1/maven-jaxb22-plugin-0.8.1.jar
urls[7] = file:/home/hudson/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.5-b10/jaxb-impl-2.2.5-b10.jar
urls[8] = file:/home/hudson/.m2/repository/com/sun/xml/bind/jaxb-xjc/2.2.5-b10/jaxb-xjc-2.2.5-b10.jar
[FATAL ERROR] Container realm = plexus.core.maven

[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] javax/xml/bind/ValidationEventLocator
[INFO] ------------------------------------------------------------------------
[INFO] Trace
java.lang.NoClassDefFoundError: javax/xml/bind/ValidationEventLocator
	at com.sun.tools.xjc.reader.internalizer.DOMForestScanner.scan(DOMForestScanner.java:84)

It seems that the maven-jaxb2-plugin –we use for JAXB to generate java sources for a certain schema–suddenly triggers a (missing or incompatible?) dependency. The weird thing is the “suddenly” part, since the last commit was done on Friday after which the normal continuous builds ran fine and the even the nightly build on midnight Friday to Saturday. We use Maven to take care of these things right? Always have a known set of dependencies, each part of the process?

Ofcourse at the time things didn’t seem so obvious. First thing I wondered, could it have something do to with a Java 5 vs 6 problem? As a matter of fact, on our project we’re trying to go to Java 6 after a long time, but I’m the only one which already upgraded my local developer machine to work with JDK6 for a while and see if nothing strange happens. You know, keeping a lot of files from your workspace in a special “DO NOT COMMIT” working set or hidden from view, in order to prevent it from prematurely ending up in version control

So naturally, I first explored all changesets leading up to the failed build to see if I by accident committed a file I shouldn’t have. But I couldn’t find anything.

Next I thought: Could Hudson be configured incorrectly last week by a parting co-worker (which had a somewhat unofficial ownership of our CI infrastructure until he left) which now surfaced, because e.g. a server was restarted or Hudson was upgraded? Checked a few things out: our  Hudson startup.sh looked like

/usr/java/latest/bin/java -jar hudson.war --httpPort=9090 --ajp13Port=9091 --deamon --logfile=hudson.log &

where /usr/java/latest/bin/java pointed to “Java(TM) SE Runtime Environment (build 1.6.0_24-b07)” while normal “java” on the command-line pointed to “Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_22-b03)” . Could it be that the “latest” directory pointed earlier to Java 5 en (by some mysterious upgrade on the server) now suddenly links to Java 6. Could the Java version we start Hudson with affect the Java version we use for our Hudson jobs?

I don’t hope so! Consequently, I checked the Hudson configuration itself: what kind of JDK’s did we configure? Luckily, only one: named “1_5_0″ with JAVA_HOME pointing to “/usr/java/default”. But again not clear from the path to what Java version it points, until I figured out this symlink leads to “/usr/java/jdk1.5.0_22″. Good, JDK 5 should be used as default for all our Maven builds. To verify I added a 2nd bogus JDK entry in the Hudson configuration, so I was able to explicitly choose it in the job configuration – and after triggering a new build I was sure Hudson was configured correctly JDK-wise. Man, how one can digress!

So let’s take a look at the offending library at hand. From the stacktrace one can spot the maven-jaxb2-plugin artefact from the group org.jvnet.jaxb2.maven2. Did something happen to this dependency itself? I opened up our Nexus console, searched for it and saw the screen below:

Wait a minute! Why do we have two versions of this plugin? In Nexus I opened up the tab Artefact Information on the 0.8.1 version and saw that the uploaded date listed last Saturday 28th of Jan. I know we shouldn’t have more than one version of this plugin in Nexus. Did a small check on the JIRA release notes page of this plugin…

Well then. So a new release of the plugin was published online last Saturday and the nightly build triggered the download of a new version 0.8.1 – which seemed to be compiled against Java 6 now which caused the java.lang.NoClassDefFoundError. This naturally lead me to the offending part in our pom.xml:

            <plugin>
                <groupId>org.jvnet.jaxb2.maven2</groupId>
                <artifactId>maven-jaxb2-plugin</artifactId>
            </plugin>

Argh. No version!

Changed it into:

            <plugin>
                <groupId>org.jvnet.jaxb2.maven2</groupId>
                <artifactId>maven-jaxb2-plugin</artifactId>
                <version>0.8.0</version>
            </plugin>

I committed the pom.xml and triggerd a manual build – and… it worked. Ofcourse.

In conclusion there are some lessons to be learned I think:

• When debugging a problem, always try to reason about possible paths which come to mind to follow first and don’t get hung on your very first idea which pops up. This got me I think sidetracked too long on finding out whether or not I accidentally committed Java 6 files or whether or not Hudson got accidentally misconfigured into not using Java 5 anymore somehow. Troubleshooting skills are highly dependent of previous experience in a field and ofcourse, if ever a similar problem arises I’d be able to recognize it faster now.
• Write it down – so a collegue or yourself can find it back more easily later on.
• Explicity set dependency versions in your pom.xml! Don’t blame Maven for everything

Does anyone else have a troubleshooting-tale of going in the wrong direction?

Original posting: http://tedvinke.wordpress.com/2012/02/01/why-does-my-maven-build-fail