Improve exploded structure experience for efficient deployments

[snicoll]

The reference guide has a section on unpacking the executable jar to get extra boost. It also has an additional hint for bypassing the bootlader with the impact on losing classpath ordering.

Our work on AppCDS (see spring-projects/spring-framework#31497) has shown that a predicable classpath has an impact on how effective the cache is going to be. Investigating a bit more, it looks doable to provide the above as a first class concept.

Here is a proposal that is hacking layertools with an additional extract2(sic) command: https://github.com/snicoll/spring-boot/tree/gh-38276

Ignoring the fact that this does use layertools for convenience, you can execute the following on any repackaged archive:

java -Djarmode=layertools -jar target/my-app-1.0.0-SNAPSHOT.jar extract2 --destination target/app

This creates a directory with the following structure:

target/app/
├── application
│   └── my-app-1.0.0-SNAPSHOT.jar
├── dependencies
│   ├── ...
│   ├── spring-context-6.1.0-RC2.jar
│   ├── spring-context-support-6.1.0-RC2.jar
│   ├── ...
└── run-app.jar

You can then run the app as follows (assuming you're in the same directory as the previous command):

java -jar target/app/run-app.jar

The run-app.jar has the following characteristics:

• It starts the application class directly (no bootlader used)
• The manifest defines a classpath with the same order as classpath.idx, with application/my-app-1.0.0-SNAPSHOT.jar being in front

my-app-1.0.0-SNAPSHOT.jar has some manifest entries of the original jar so that package.getImplementationVersion() continues to work.

The work on this prototype has led to a number of questions:

• Layertools provide a Command infrastructure and several utilities related to extracting/copying that are not specific to layers. If we want to go the same route with a different jar mode, a significant number of classes should be copied/reimplemented. Perhaps a single jar with a public API and sub-packages for layertools and this new mode could be an option?
• This work could potentially apply to layers themselves. Rather than having dependencies/BOOT-INF/lib and a layer for the bootloader, we could use the same exact structure. Or perhaps the structure above could become a layer configuration (where all libs go to dependencies).
• It's hard for run-app.jar to have sensible File attributes. The command tries to respect the file attributes of files it extracts from the repackaged archive, but run-jar.jar is created on the spot.

We have confirmed that with the prototype, AppCDS is effective (close to 95% classes loaded from the cache).

[sergmain]

About performance - In my project a data exchange between agents conducted via zip files. In one case I had 1k+ zips inside other zip and preparing data for agent by unziping all archives had taken minutes.
After switching to com.google.jimfs the time of preparing reduced to hundreds milliseconds.
So spending additional memory you can achieve almost zero time for unpacking without making any changes in structure.of archive.
You can see a general concept here
https://github.com/sergmain/metaheuristic/blob/master/apps/commons/src/test/java/ai/metaheuristic/commons/utils/ZipUtilsTest.java

pom dependency

<dependency>
    <groupId>com.google.jimfs</groupId>
    <artifactId>jimfs</artifactId>
    <version>1.2</version>
</dependency>

repo is here - https://github.com/google/jimfs

[snicoll]

@bclozel and I had a bit of brainstorming on this one and he raised a use case that could become problematic. The solution above ignores the Spring Boot launcher on purpose to make the startup as straightforward as possible. However, it currently does not allow to augment the classpath, which is something that can be problematic. I don't know if this is something we need to take into consideration, but the POC could be adapted accordingly.

[snicoll]

The PoC has been updated to provide the ability to include additional libraries. This is typically used by buildpacks, for instance:

$ java -Djarmode=layertools -jar my-app.jar extract2 --destination target/app --additional-jars jar1.jar,jar2.jar

This copies jar1.jar and jar2.jar available in the current directory to target/app/ext. The two jars are added at the end of the classpath in the manifest of run-app.jar.

One decision could help move the POC into something that can be reviewed. If the above is working for buildpacks, perhaps this could become the default mode of operation for layer tools as well. In other words, merging extract and extract2.

The structure above reuse the concept of "layers" to nicely separate concerns, except it does not expand the structure of the far jar (i.e. dependencies has libraries directly nested rather than BOOT-INF/lib). One could argue it is completely unnecessary and a simple lib directory with all jars (including the generated jar for the application) could be enough.

Merging the two means we need to bring back the launcher, perhaps as an opt-in?

Keeping things separate would be less disturbing. However, the argument that buildpacks would work with the new mechanism, and has to for optimal CDS performance, means two mode of operations that might not be desirable. There are also a lot of infrastructure in jarmode that would either need to be duplicated or abstracted.

I am happy to improve the proposal based on what the team decides.

[wilkinsona]

I spent some time looking at this last week and we discussed it a bit yesterday. The high-level conclusion was that we need to combine support for layers and CDS in a single command. This is necessary as the former needs to know about the latter so that the runner jar can go in the most frequently changing layer.

At the moment, I'm imagining a single command with two options – one for CDS and one for layers – that can each be used individually and that can also be used together. There's quite a lot to figure out in terms of how we get from where we are now to where we want to be in a non-breaking manner. We hope that we might be able to look at this in March but there are other, higher priority items that require our attention in the 3.3.0-M2 timeframe.

[snicoll]

I spent some time looking at this last week and we discussed it a bit yesterday. The high-level conclusion was that we need to combine support for layers and CDS in a single command.

Yes. I guess I should have dumped my thoughts somewhere as I reached the same conclusion. Considering the buildpacks use case, I see it as a continuation of what we've started when we introduce layertools back then. Buildpacks still need all the features that we've built so far, but with a mode of operation that allows to further tune the application before final packaging.

Given the classpath cannot be extended further, a mode of operation could be that run-app.jar is created as part of repackaging the archive, so that its timestamp matches, rather than creating it on the fly when we extract the archive.

Then, we need to review layers.xsd. So far, extract has taken the route of including all artifacts that are present in the archive, but this is no longer the case. Instead, there are libraries, application classes and resources, and a "loader". The loader can either be the Spring Boot loader or this new run-app.jar mode of operation. It's interesting to see that layers.xsd list the resources to include in a layer, not where they end up: the current mode extracts the resource as is in a directory that matches the layer name. This gives us the flexibility to know in advance how many layers there are, how many directories we need, and how to structure run-app.jar upfront.

With buildpacks being updated upfront to support the new infrastructure, we could imagine a mode of operation where the build plugins introduce a flag that defaults to this run-app.jar mode rather than the classic mode we had so far. This would obviously force those who are crafting layered docker images to manually set a property when upgrading.

There are a ton of things that I overlooked, but I am happy to dig further based on the team's guidance when it gets to this. Thanks!

[wilkinsona]

Given the classpath cannot be extended further, a mode of operation could be that run-app.jar is created as part of repackaging the archive, so that its timestamp matches, rather than creating it on the fly when we extract the archive.

We'd like to maintain the current model where the archive produced by the build plugins is the single source of truth. It can then be run as-is using java -jar or it can be further processed to produce a deployment-specific arrangement as it is today. We don't want the build plugins to produce the runner jar (and presumably the directory of dependencies along side it) directly as our feeling is that it will create too much confusion and introduce too much overlap with things like Gradle's application plugin and Maven's assembly plugin.