Working group: Class loading in Quarkus

[dmlloyd]

Greetings fellow Quarkoids!

I'd like to propose the formation of a working group in order to address some long-standing class loading architectural issues within Quarkus.

Current Context and Problem Statement

At the time of this writing, the Quarkus codebase contains between 15 and 20 class loader implementations. These implementations serve a variety of purposes:

• Exposing hidden API methods
• Setting up a (different) execution environment for a variety of application launch modes
• Working around problems in upstream projects
• Customizing parent delegation in a variety of situations

Each of these class loaders has a different behavior; some are outright incorrect. Additionally, we have a number of problems relating to specific implementations. These problems cover such areas as concurrency, virtual threads, delegation problems, and more.

The ad-hoc approach to class loading served us to get the Quarkus project off the ground quickly. However, at this stage of project evolution, I think it would be a good idea to take a top-down look at what our class loading needs are, and reevaluate the architecture. I think it would be particularly wise to consider the past (what did or did not work well for WildFly), the present (what does or does not work well in our existing implementations, what are GraalVM's needs), and the future (supporting JPMS, FFM, Leyden, and other future considerations).

Proposal

I'd propose we approach this in phases. The first phase (which is already underway) would involve reducing the surface area of the problem by looking through the various existing class loader implementations and cleaning them up, documenting them, and/or removing them from the code base if they are no longer needed. See #43726 for a top-level issue covering this work.

The second phase would be a top-down analysis of how class loading should ideally work at both build and run time. This is where a more formal working group would be of most use. In this phase, we would consider what performance and behavioral characteristics are expected at both build time and run time.

A non-exhaustive list of things to consider:

• Virtual threads and pinning
• Concurrency behavior
• Isolation behavior, including service loading delegation
• JPMS module support (to what degree? can we future-proof?)
• Memory-mapped archives - feasible or not? Is this a viable concurrency solution?
• In-IDE behavior
• Class loading for various kinds of launch modes
• Native image class loading
• Security (JAR signing?)
• Class loader lifecycle/closure
• Relationship to Maven and Gradle
• Relationship to dependency artifact resolution (tighter integration?)
• Relationship to test class loading (see Working group proposal: Test classloading #41867, also consider: split package support?)
• Layering behavior: build time, run time deps, run time app
• Development mode and partial reloading of CL environment
• Any API promises that our current class loaders are making that need to be upheld or deprecated
• Lessons learned (positive and negative) from the WildFly implementation

The third phase would involve prototyping some kind of centralized class loading system which would be adaptable to all of the applicable scenarios above.

The final phase would be integration of the new class loading system and reorganization of the existing code around it.

Expected outcome

The expectation would be that a variety of problems which relate to class loading would be solved, and that future improvements (particularly those relating to JDK restrictions around FFM integration, reflection, and serialization, as well as JDK improvements such as the Project Leyden efforts) would be more possible and/or simpler compared to today.

There is also an expectation that class loading performance would be on par with today, if not better.

As a knock-on benefit, whether or not JPMS support is included, we should be able to provide users with more useful stack traces which include artifact IDs and versions for dependencies.

Related issues (non-exhaustive)

• Class loader cleanups (phase 1) #43726
• RunnerClassLoader - Some classes are loaded multiple times #42874
• Some classes loaders are not ARM compatible #43415
• Mitigate concurrent classes definition in RunnerClassLoader #43022
• Load Testcontainers parent first and start the containers with the Platform CL as the TCCL #42560
• Fix ClassLoader leak in KeycloakDevServicesProcessor #42338
• Replace read/write lock in JarResource to avoid virtual threads pinning #40942
• Netty finalizers load classes from closed class loaders #41607
• Class Loader still used while it's supposed to be closed #41233
• More classloaders cleanup #40557
• Remove deprecated classloader helpers #40564
• Improve loading of classes in RunnerClassLoader #40033
• Most class loader implementations aren't parallel-capable #39952
• Discontinue support for *JarRequiredBuildItem #44518

[cescoffier]

How does it differ from the working group https://github.com/orgs/quarkusio/projects/30? Should it be merged?

CC @holly-cummins

[holly-cummins]

I think it's reasonable to keep these separate, in the interests of having smaller scopes. https://github.com/orgs/quarkusio/projects/30 covers test classloading specifically, whereas the proposed WG has the broader remit of "all classloading."

However, since there's a substantial overlap, I'd put in a plea to sequence the working groups, and not start serious work on the broader classloader rewrite until https://github.com/orgs/quarkusio/projects/30 is mostly done. Otherwise there's likely to be a lot of rework-cost.

[dmlloyd]

I agree, this is a separate but overlapping effort. And +1 to getting the test stuff done sooner. Having the test class loading rework complete first should simplify this effort, at least to a degree.

[dmlloyd]

Dynamic module loading with JPMS support does seem possible; here's a fairly minimal prototype modeled after the jboss-modules philosophy which implements it and can launch itself: https://github.com/dmlloyd/modules2

[jbonofre]

I'm investigating to support "dynamic loading" with classloaders management.
I started Quaraf (similar to Apache Karaf services but powered by Quarkus).

The idea is to focus on Karaf key features (hot deployment/deployers, classloaders management, service registry/discovery, ...) by leveraging Quarkus.

So, happy to participate in this discussion. I will share on https://github.com/jbonofre/quaraf soon.

[triceo]

Perhaps off-topic here, but was caused by a recent classloading change:
#44283

[franz1981]

I was looking at https://github.com/quarkusio/quarkus/blob/5a2cb66525dc197b2c71ba8bf72eef4e298ef370/independent-projects/bootstrap/runner/src/main/java/io/quarkus/bootstrap/runner/SerializedApplication.java and:

1. I would like to avoid the big amount of single-threaded work performed there to produce the maps/sets etc etc which are used by the runner class loader
2. I would like to test if mmap is beneficial, without assuming it

The reason for 2 is because many database (e.g. levelDB) has moved away from it for good reasons:

1. mmap would like pages to "stay still" and not causing to be read from disk multiple times. And you would like sharing these read pages (across multiple processes). Sadly if you have a container and you won't put the classes in a shared layer, this is not going to happen, and each container instance will read the same data again and again, defeating the purpose of mmap i.e. caching.
2. since OS page cache in the kernel is outside of the application control, the LRU policy could be not as good as we want and "noisy neighbours" can cause OS pages re class loading to be "evicted" and we risk to read them N times from disk even during the lifecycle of the same process. This "transparent" behavior is a known cause of so-called long time to safepoint - which usually break badly the JVMs (see https://groups.google.com/g/mechanical-sympathy/c/LFrJPhyVOJ4)
3. given 1 and 2 if we optimize for NOT sharing, having DIRECT_IO will give much more control on RSS/memory usage and would save copying from disk -> OS pages just performing a single copy when needed (likely over the same tmp off-heap buffer), just to define/link classes. This would allow too to release such buffer when not needed, enabling RSS to be lowered (we need malloc autotrimming enabled for such, since it is not what glibc malloc does by default while you free vs it AFAIK)
4. munmap does something nasty with locks which slow down process exit (not nice, for the JVM, mostly) - see https://elixir.bootlin.com/linux/v6.11.8/source/mm/mmap.c#L3401-L3452 if I got it right

In short; we should think about the use case (now and in the future) where we expect it to be beneficial, before designing things around its strenghts and getting bitten (when it's too late) by its pain points.

[franz1981]

I see, so you mmap'd the dat file itself - re the classes binaries, these are NOT in the mmaped file, right?
I believe what @dmlloyd want to do it about these

[geoand]

Correct, just the dat file

[dmlloyd]

My thought, at least for one launch mode, is to have one large JAR with the whole application in it, possibly containing other ("stored", not compressed) JARs whose classes are also not compressed, and mmap the single JAR at once. The resource loader would then return sliced buffers when classes and resources are loaded. Finally we'd madvise it away after startup completes.

But, since we would hide it all behind the resource loader API, the class loader is not tied to one strategy, which will let us compare many approaches.

[Sanne]

I would like to avoid the big amount of single-threaded work performed there to produce the maps/sets etc etc which are used by the runner class loader

Right, we discussed this 6+ months ago :) Should be straightforward, just need someone with some spare time™ to want to have some fun, and it would be clearly beneficial on its own (irrespective of the rest of the discussions here).

if you have a container and you won't put the classes in a shared layer, this is not going to happen, and each container instance will read the same data again and again, defeating the purpose of mmap i.e. caching.

As you well know that's totally my intention to make sure that at least a good chunk will be stored in a common layer, shared across applications - including different applications but with the assumption that they run the same Quarkus version. This might not necessarily be the only way of running a Quarkus application but it should be the strategy we optimise for.
But we'll never be sharing the entirety of the classpath, so I'd like to see this approach applied tactically on the core layer, but not (necessarily) on all classloaders.

.. and we risk to read them N times from disk

TBH I don't see this as a major issue. Saving memory and sharing memory should be the priority even over slightly slower bootstrap; and being in a position to potentially share memory in several cases is better than not sharing such memory just because it might not always be effective.

.. having DIRECT_IO

Sure, that's interesting and perhaps better.

[Sanne]

My thought, at least for one launch mode, is to have one large JAR with the whole application in it, possibly containing other ("stored", not compressed) JARs whose classes are also not compressed, and mmap the single JAR at once. The resource loader would then return sliced buffers when classes and resources are loaded. Finally we'd madvise it away after startup completes.

@dmlloyd +1 But could I have two "jar"s ? I'd like to have a single blob for the bootstrap classloader, and one for the rest of the application (user code + extensions). The idea being that multiple, different Quarkus applications would be able to share the boostrap layer; this layer would potentially be combined with the jdk's base modules, and optionally pre-trained for Leyden, etc..