Use the same classpath ordering for an exploded jar and an archive

[hengyunabc]

demo.zip

Print classloader urls in main method:

	public static void main(String[] args) {
		URLClassLoader classLoader = (URLClassLoader) Thread.currentThread().getContextClassLoader();
		for(URL url : classLoader.getURLs()) {
			System.err.println(url);
		}
		SpringApplication.run(DemoApplication.class, args);
	}

• Run as jar

mvn clean package -DskipTests
$ java -jar target/demo-0.0.1-SNAPSHOT.jar

output:

jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-boot-starter-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-boot-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-context-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-aop-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-beans-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-expression-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-boot-autoconfigure-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-boot-starter-logging-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/logback-classic-1.1.11.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/logback-core-1.1.11.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/jcl-over-slf4j-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/jul-to-slf4j-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/log4j-over-slf4j-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/spring-core-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/snakeyaml-1.17.jar!/
jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/slf4j-api-1.7.25.jar!/

• Run in directory

cd target
unzip demo-0.0.1-SNAPSHOT.jar -d demo
cd demo
java org.springframework.boot.loader.PropertiesLauncher

output:

file:/private/tmp/demo/target/demo/BOOT-INF/classes/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/jcl-over-slf4j-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/jul-to-slf4j-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/log4j-over-slf4j-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/logback-classic-1.1.11.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/logback-core-1.1.11.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/slf4j-api-1.7.25.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/snakeyaml-1.17.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-aop-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-beans-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-boot-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-boot-autoconfigure-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-boot-starter-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-boot-starter-logging-1.5.3.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-context-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-core-4.3.8.RELEASE.jar!/
jar:file:/private/tmp/demo/target/demo/BOOT-INF/lib/spring-expression-4.3.8.RELEASE.jar!/

---

The order of classLoader urls is very important.

When run app in exploded directory, the order of jars is different.

I recommend add a index file BOOT-INF/INDEX.LIST to save the order of jars.

See also:

• http://docs.oracle.com/javase/7/docs/technotes/guides/jar/jar.html#Index_File_Specification
• When the spring-boot-maven-plugin performs the repackage goal the POM order of the libraries is not maintained #8397

[almeidaah]

I see, i think is important too. But why need to run the exploded class? When you run, you run the jar artifact, right?

[hengyunabc]

@almeidaah

When run as fat jar, the jar url looks like :

jar:file:/private/tmp/demo/target/demo-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/slf4j-api-1.7.25.jar!/`

It is not the standard jar file url, will have compatibility issues. So we run spring boot app in an exploded directory.

[wilkinsona]

It is not the standard jar file url, will have compatibility issues

What compatibility issues have you seen? In the vast majority of cases, the nested jars do not cause any problems. When they do, you can configure specific dependencies to be unpacked to a temporary directory on launch. In other words, while it's not an unreasonable thing to do, there should be no need to unpack the jar as you are currently doing.

[hengyunabc]

@wilkinsona
Thanks, I know the unpacked to a temporary directory on launch feature.
But the user's scene is complex, some code will get all ClassLoader urls, process themself.

[wilkinsona]

I've reworked the issue title as I don't want to jump straight to a particular implementation. The problem's also more complex than it would appear to be at first. One reason for exploding an archive is to add extra jars to the classpath (CloudFoundry's Java buildpack does this, for example) at that point any list that orders the classpath becomes stale and we'd need to decide what to do. Some options include:

1. Fail
2. Place any extra jars at the start of the classpath
3. Place any extra jars at the end of the classpath

[brenuart]

Isn't enough to simply sort the entries alphabetically to have a predictable classpath construction?
I would expect BOOT-INF/classes to always appear first, followed by entries in BOOT-INF/lib sorted on their name. This would give a classpath similar to what Maven and most servlet container do (afaik).

The same logic would apply when running on an exploded jar. If people want to add extra jars, then they would know they have to pay special attention to their names if they want them to appear first/last in the classpath.

Regarding the additional paths specified by loader.path, I would add them first just like the current behaviour. To me, this feature is a way to override part of what is in the original fat jar.

Side question: is there any guarantee today that BOOT-INF/classes ALWAYS appears before BOOT-INF/lib ?

[wilkinsona]

Isn't enough to simply sort the entries alphabetically to have a predictable classpath construction?

No. As described above, even with a determined order, we need to preserve that order when a jar is exploded and also allow jars to be added somehow.

I would expect BOOT-INF/classes to always appear first, followed by entries in BOOT-INF/lib sorted on their name. This would give a classpath similar to what Maven and most servlet container do (afaik).

Maven orders the class path based on the order in which dependencies are declared in the pom. I believe Gradle does something similar.

is there any guarantee today that BOOT-INF/classes ALWAYS appears before BOOT-INF/lib ?

No. It's determined by the order of the entries in the jar file, or, if it's been exploded, by the ordering of the files on the filesystem.

[brenuart]

we need to preserve that order when a jar is exploded

The loader could sort them when building the classloader.

Maven and the declaration order

You are correct, my bad.

BOOT-INF/classes first

This is bad news I must say. Bottom line is the classpath is seems to be unpredictable (or depends on external factors like the type of filesystem (ordering, case sensitive, etc).

I'm concerned about the way SpringBoot loads the application configuration properties (or other type of resources). If nothing exists on the current directory, it tries to load classpath:application.properties... What if such file exists in the root of some used library? My feeling is application classes (at least) should always appear first.

This seems to be the case when running on the fat jar (not expanded) - but not by design. I have the impression the zip/jar entries are always returned in alpha order and it works because BOOT-INF/classes appears before BOOT-INF/lib (because the jar is indexed?)... I may be wrong tho.

[wilkinsona]

The loader could sort them when building the classloader.

Once again, it's not that simple. Firstly, if the loader sorted them it would then be using a different classpath order from in your IDE. Secondly, we need to be able to add jars somehow and be deterministic about where they appear in the list.

What if such file exists in the root of some used library?

A library shouldn't contain an application.properties file. Even if the application's own application.properties file was guaranteed to appear first on the class path, there's no requirement for an application to have such a file. At that point, the library is polluting the application's configuration.

[brenuart]

Firstly, if the loader sorted them it would then be using a different classpath order from in your IDE.

That's no the point. The idea is to have a predictable order. Today I can't explain my developper what the classpath will be.

Secondly, we need to be able to add jars somehow and be deterministic about where they appear in the list.

I don't clearly see the problem here (maybe I'm blind)...
I don't understand why sorting the entries found in directories (would it be a folders inside the fat jar or on the filesystem) affects the ability to add extra jars. Sorting on the name gives a predictable classpath ordering and allows people to rename their jars if they want them to be inserted in a particular position. It also reduces the impact of the filesystem environment and/or the order entries are added in the fat jar at construction time (which may affect the order they are discovered at runtime).

Anyway, as you understand, the point is we don't feel comfortable with how the classpath is constructed today as it seems its order may change in unpredictable ways (or because of factors not under our control).

A library shouldn't contain an application.properties file.

I agree - but you don't always control external libraries... application.properties was an example but this applies to any kind of resource the application is about to load from the classpath. Today, there is no way to guarantee the resource of the application isn't shadowed by one from an external library. This is our biggest concern.

(sorry to insist on the subject, but classpath construction is a recurrent question/issue here and I have not been able so far to answer with an "acceptable" explanation :(

[wilkinsona]

Firstly, if the loader sorted them it would then be using a different classpath order from in your IDE.

That's no the point. The idea is to have a predictable order.

You would be happy for your application to use one order for its classpath when run in your IDE and one when it's launched using one of Boot's launchers (either as a fat jar or as an exploded fat jar)?

I don't understand why sorting the entries found in directories (would it be a folders inside the fat jar or on the filesystem) affects the ability to add extra jars.

We can't sort things alphabetically. That isn't what happens in your IDE when you run an application's main method and it isn't what happens when you use mvn spring-boot:run or gradlew bootRun. We quite deliberately try to build a fat jar using the same ordering as your build system does. That means that your application's tests and your application's jar will hopefully be run with the same class path.

Given that we can't sort things alphabetically, it becomes harder to know where on the class path any extra jars that have been added to an exploded BOOT-INF/lib directory should be placed. This is the problem that I've described above. It's not insurmountable, but it means that this isn't as simple as you suggest.

Anyway, as you understand, the point is we don't feel comfortable with how the classpath is constructed today as it seems its order may change in unpredictable ways (or because of factors not under our control).

I believe that's only a problem if you are comparing running a fat jar with java -jar and exploding a fat jar and running it using the launcher. If you don't explode your fat jar then the classpath order is entirely predictable and is governed by your build system. Note that using the build system's ordering of the class path to oder the entries in the jar means that you'll get the same order when running the app with the build system (mvn spring-boot:run or gradlew bootRun) and java -jar.

[brenuart]

You would be happy for your application to use one order for its classpath when run in your IDE and one when it's launched using one of Boot's launchers (either as a fat jar or as an exploded fat jar)?

It is already the case: as far as I can tell Eclipse puts the application classes first, followed by the dependencies in the order they were declared in the pom. So does surefire during the tests.
This would be in favour of having BOOT-INF/classes always in front of the rest.

For what concerns the rest of the class path (libs), I understand your point and it is totally understandable you want to somehow "delegate" the ordering to the build system.
So, if I get it right, the classpath order is the order entries are added into the ZIP. And this order matches the order dependencies are returned by the maven dependency mechanism. Correct?

The zip-order is obviously lost when exploding the archive on the filesystem - leading to a potentially different classpath. Therefore your note saying the "problem" affects only those running the app on an exploded jar.

The zip-order may also be lost if the zip is altered later (uncompress/recompress to add stuff, edit to modify a resource, etc). You may end-up with a new zip with entries in a different order and therefore a different classpath. Such operation should not be recommended. Correct?

[wilkinsona]

It is already the case: as far as I can tell Eclipse puts the application classes first, followed by the dependencies in the order they were declared in the pom.

I don't think it is the case. Assuming you're using m2e in Eclipse, its Maven that's ordering the class path. The same ordering is using by mvn spring-boot:run and is reflected in a fat jar too.

This would be in favour of having BOOT-INF/classes always in front of the rest.

I disagree. BOOT-INF/classes should not receive any special treatment in Boot. Instead, we should honour whatever class path order the build system gives us.

I mean does the classpath respect the order dependencies are listed in my pom

Yes.

This remains true unless the zip is uncompressed/recompressed in the meantime for whatever reason - which can lead to a different order of the entries. Ideally one should avoid such (dangerous) operation.

We know that and we agree. That's why this issue hasn't been declined.

[brenuart]

Thanks for all the information. This makes things clearer.
Bottom line is: classpath at runtime is the one determined by Maven (or should be).

About Eclipse classpath, it seems to be made of the following in that order:

• JRE system library
• Application sources (and classes)
• Maven Dependencies (in the order resolved by Maven)

About Surefire classpath, it is made of the following:

• JRE system library
• Application test-classes
• Application classes
• Maven Dependencies (in the order resolved by Maven)

Application classes appear before libraries in both cases.

[brenuart]

Sorry to come back on this, but after having a look at the Repackager it seems that application classes are written BEFORE any libraries - and therefore BOOT-INF/classes always appear in front of BOOT-INF/lib.

Cfr. https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-tools/spring-boot-loader-tools/src/main/java/org/springframework/boot/loader/tools/Repackager.java#L252-L267

1. write unpack libraries
2. write application classes
3. write librairies
4. write loader classes

Since you said that BOOT-INF/classes should not deserve any special treatment, I was wondering if we can rely on the order these four items are written or if it should be considered as implementation details?

[wilkinsona]

Thanks for taking a deeper look.

I think the separation of 1 and 3 is a bug as configuring a library to be unpacked from the nested jar on launch shouldn't affect its position on the class path, particularly as it'll mean that order is no longer aligned with what Maven has given us. Can you open an issue for that please?

Since you said that BOOT-INF/classes should not deserve any special treatment, I was wondering if we can rely on the order these four items are written or if it should be considered as implementation details?

I was wrong there. Sorry. My recollection was that we got the whole class path (local classes and jar dependencies) from Maven and then wrote them out in whatever order we got them in.

Right now, I'd consider the ordering of application classes vs libraries to be an implementation detail, although it's one that we're unlikely to change, particularly in a maintenance release. I don't think it would be unreasonable to offer some stronger guarantees about that ordering. We could also verify that there's some consistency between Maven and Gradle. Please open another issue for that if it's something you'd like to see. This issue really needs to remain focused on the differences that can occur when a fat jar is exploded.

[brenuart]

Done.
Thanks for all the info and sorry for the hijacking of the thread...

[ianfairman]

We've just spent a couple of days tracking why a spring boot application built on Linux using Jenkins ran slower than the same application built on a Windows 10 PC. We've not got a complete understanding of the issue but it's clear that classpath ordering is part of it. A colleague unzipped and re-zipped the slow running app and the problem went away and the only thing that had changed was the ordering of the files in the zip.

I wouldn't be so worried about the ordering of jar files if it weren't for the fact that some classes are defined in multiple jars (for example AOP and JPA classes) which is very poor practice but which we have to live with. The jars to which I refer are dependencies of spring boot.

Are there plans to tackle the general classpath ordering issue?

[wilkinsona]

Can we please avoid derailing this issue any further. It is specifically interested in preserving class path ordering between a zipped archive and an exploded archive.

The jars to which I refer are dependencies of spring boot.

We use the Maven duplicate finder plugin in our starters to ensure that there is minimal duplication of classes in the starter's dependencies. Unfortunately, eliminating duplication entirely is sometimes impossible due to packaging of third-party code.

Are there plans to tackle the general classpath ordering issue?

As far as we know, there's no general class path ordering issue. The order of the jars in the BOOT-INF/lib directory of a fat jar is determined by Maven or Gradle.

@ianfairman If you have encountered something that as it odds with what I've said above, please open a new issue that is focussed on a specific, clearly described problem.

[thiagohmoreira]

Hello everyone,

I just want to add a small piece of information. I've ran into issues with the classpath loading order after following this guide: https://spring.io/guides/gs/spring-boot-docker/ (due to some thridparty dependencies I can't fix)

My app runs fine from the boot's JAR, but fails to start on the Docker image due to the different classpath loading order. I can of course workaround this by making my Docker image from the fat jar and not the unpacked version, but I would suggest to put at least some mention about this potential issue on that.

[wilkinsona]

Thanks for the suggestion, @thiagohmoreira. I've opened spring-guides/gs-spring-boot-docker#71.

[wilkinsona]

Don't we need some Gradle plugin changes for this too?

[philwebb]

I've opened #19847 for the Gradle part.