I suddenly had the feeling that ther’s something fishy with the maven-eclipse-plugin. And in deed, a look into the ~/.m2/repository/org/apache/maven/plugins/maven-eclipse-plugin proved me right. There is a brand new version 2.6 of the plugin.

This new version automagically adds projects from the workspace rather than JARs from the local repository (which is cool, basically). However, that caused Eclipse to load the wrong JARs: One project depended on sl4j 1.3, another one on 1.5 and the other project. Maven perfectly resolves the version of this transient depenedency, while Eclipse picks one by chance. Gotcha#1!

The first warning was caused by maven-compiler-plugin beeing configured to compile for Java 1.5 while only a 1.6 JDK was available. Gotcha#2!

Hopefully this post will save other people from being as clueless as I’ve been

Today, I stumbled upon a classpath related problem – once again. As I doubt that I am the only one to ever face this problem, I want to share a short shell script that came to the rescue.

But first, what was the problem? After adding some additional dependencies to our POM – quite carelessly I have to admit – our application started sending mails without subject, sender address and messed up special characters. Interestingly enough, I didn’t touch the mail part at all. I added Apache CXF dependencies, i.e. web service stuff. So what was wrong?

CXF comes with quite a lot of transitive dependencies (that should be declared optional, I’d say), including some Geronimo Spec jars. One of them was org.apache.geronimo.specs:geronimo-javamail_1.4_spec that superseded classes from javax.mail:mail. A simple exclusion fixed the problem.

However, I wanted to be prepared for the next time as this tends to happen quite regularly with a growing number of transitive dependencies. There are packages with same content and different id, (e.g. org.mortbay.jetty:servlet-api and javax.servlet:servlet-api), artifacts that contain subsets of other artifacts (e.g. org.springframework:spring contains everything from org.springframework:spring-web), and artifacts that should be replaced in favor of others (e.g. slf4j instead of commons-logging.

That’s why I wrote this little bash script:

#!/bin/bash
for lib in `find . -name '*.jar'`; do
for class in `unzip -l $lib | egrep -o '[^ ]*.class$'`; do
class=`echo $class | sed s/\\.class// | sed s/[-.\/$]/_/g`
existing=$( eval "echo $CLS_${class}" )
if [ -n "$existing" ]; then echo "$lib $existing"; fi
eval CLS_${class}="("${lib} ${existing}")"
done
done | sort | uniq -c | sort -nr

These few lines of code print packages and the number of their common classes. Sample output is:

stf@crabman:/path/to/my/webapp$ duplicates.sh
127 ./WEB-INF/lib/aspectjweaver-1.5.3.jar ./WEB-INF/lib/aspectjrt-1.5.3.jar
117 ./WEB-INF/lib/spring-2.0.8.jar ./WEB-INF/lib/spring-web-2.0.8.jar
83 ./WEB-INF/lib/commons-beanutils-1.7.0.jar ./WEB-INF/lib/commons-beanutils-core-1.7.0.jar
38 ./WEB-INF/lib/servlet-api-2.3.jar ./WEB-INF/lib/servlet-api-2.5-6.1.11.jar
10 ./WEB-INF/lib/commons-beanutils-core-1.7.0.jar ./WEB-INF/lib/commons-collections-3.2.jar
10 ./WEB-INF/lib/commons-beanutils-1.7.0.jar ./WEB-INF/lib/commons-beanutils-core-1.7.0.jar ./WEB-INF/lib/commons-collections-3.2.jar
9 ./WEB-INF/lib/spring-2.0.8.jar ./WEB-INF/lib/aopalliance-1.0.jar
6 ./WEB-INF/lib/commons-logging-1.1.jar ./WEB-INF/lib/jcl104-over-slf4j-1.5.0.jar

The code isn’t very fast, but makes solving a common classpath problem a lot less painful. (Note that I am using plain unzip instead of jar as didn’t have a JDK installed on this server/)

After some swearing and investigation I finally found the problem: In my code, I used ClassLoader.getSystemResource(...) to find a resource, which internally calls ClassLoader.getSystemClassloader(). However, with Maven’s Surefire plugin, the system classloader does neither contain target/classes nor target/test-classes in its classpath. Therefore, you have to make sure, that the classloader that loaded the test cases (and the tested classes) is used to get the resources.

Therefore, instead of

ClassLoader.getSystemResource(...)

always use

this.getClass().getClassLoader().getResource(...)

Problem solved!

Everything was working nicely until I carelessly added several dependencies this morning. Suddenly, my Wicket-based webapp stopped working with some strange exception. A class in Jetty called a method that was not found in the servlet API (javax.servlet). Well, rather strange, I thought. After a little online-research I found that this method was introduced in version 2.5 of the servlet API. I doublechecked with the Jetty docs and yes, Jetty 6.1 implements exactly this version of the servlet API. Well … so what’s wrong?

But then I noticed that Maven added two different versions of the servlet API to my dependencies: 2.3 and 2.5. But isn’t Maven meant for resolving such conflicts? In deed it is and it actually wasn’t Maven’s fault. While Jetty comes with its own version of javax.servlet (called org.mortbay.jetty:servlet-api-2.5 in Maven), Maven added javax.servlet:servlet-api (version 2.3) as a transient dependency.

So how to get rid of unwanted dependencies? In Maven it is possible to exclude transient dependencies. The problem is that you have to do that for every package that depends on this unwanted package. You have two choices to find out which package depends on it: you either look at the POM of every dependency and the POMs of their dependencies … or you look at the dependency tree that can be generated with Maven’s dependency plugin (mvn dependency:tree). Well, the tree-task didn’t work for me, for whatever reason. I remembered that the dependency tree was also part of the project website that can be generated with Maven’s site plugin. I did generate the project’s website (mvn site) and there was also a dependency tree … but javax.servlet:servlet-api wan’t in that tree … but in the list of transient dependencies. If I remember correctly that was were I started swearing. So it was time to manually inspect all the POMs. (For the sake of completeness: I think that the servlet API wasn’t in the dependency tree as it only contains three levels of dependencies.)

I started analysing the POMs one-by-one and found out several interesting things:

• Some POMs added junit as dependency without specifying the test-scope (it would therefore be added to the generated WAR file)
• Optional dependencies and alternatives are not defined consistently: sometimes all optional dependencies are normal dependencies, sometimes none of the alternatives is a dependency, … altogether it’s quite messy

I finally found the dependency that added version 2.3 of the servlet-api to my dependencies: commons-logging. Yes, exactly that commons-logging that is basically meant to write some lines into a file. Why does it depend on the servlet API? Because it has its own logger that is supported by commons-logging and the commons-logging POM has dependencies on all supported logging implementations:

• log4j:log4j
• logkit:logkit
• avalon-framework:avalon-framework
• javax.servlet:servlet-api

So you have to explicitly exclude the dependencies you don’t need … or you will deploy your desktop application with the servlet API. And the best thing of all: you have to do that for each and every dependency that introduces commons-logging to your project dependencies.

I then decided not only to exclude all unnecessary logging-frameworks but commons-logging all together in favour of slf4j. Unfortunately, it’s not possible to globally exclude a package all together – as Maven developers want to “protect” you from doing that – excluding commons-logging can get quite complicated. Erik van Oosten came up with a nice workaround for this topic. He announced a faked Maven repository that offers an empty jar file with version 99.0-does-not-exist for all artefacts you ask it for. So if you explicitly depend on this version, no lower versions will be added to your dependencies. And if you mark this dependency as provided, it also won’t be included in your WAR files. Yes, it’s a hack, but it’s a nice one

Besides a small problem with empty jar files that Erik is going to fix (see my comment in Erik’s blog) the whole thing works like a charm – no commons-logging, no version conflicts with the servlet API, everything is fine.

I think I still can’t answer the question whether Maven is a curse or a blessing. The only thing I can say is that Maven has its problems, but also great power. I’d say it’s very important that you really go into the topic before you use it. Maven is big, *really* big. It’s not like Ant that you can start using after reading a short tutorial. If you don’t know how Maven works it will certainly kick your ass. But if you know how to handle that beast, it really kicks ass

However, I am still not happy with the official Maven repositories. They are completely messy and Maven is not really helpful in coping with this mess. I would really love to see something similar to Debian package management: with provides, depends, suggests, recommends, meta-packages, nice updates … and a repository that is under tight control. Maybe even the new Ant subproject Ivy is a step into the right direction (I would love to see it in action together with Maven). But for the moment the only thing you can do is to have a close look on each and every POM in your tree of dependencies. Yes, it is extensive work, but it’s the only way to reduce your dependencies to a minimum … And dear Maven developers: please don’t try to prevent us from doing evil things, as we might do them anyway (like using version 99.0-does-not exist). Just give as the freedom to do what we want to do.