Class and resources discovering and loading during build / augmentation time change in 2.7

[michael-simons]

Hi there.

I'm sorry for the noise, but I'm directly paging @lukehutch and @aloubyansky here (the latter is hopefully the correct one, @gsmet ;) )

I maintain Neo4j-Migrations: https://github.com/michael-simons/neo4j-migrations

Which let's you execute Cypher scripts (resources) and Java classes against Neo4j.

I use ClassGraph to discover those resources.
Since last release I do support Quarkus in native mode. To make this possible, I moved all discovering of things into build time / augmentation. That works well with Quarkus 2.6.x (all releases).

Those classes:

https://github.com/michael-simons/neo4j-migrations/blob/0db309476f9c737b1b4674193f838f7e3c74ec75/neo4j-migrations-quarkus-parent/runtime/src/main/java/ac/simons/neo4j/migrations/quarkus/runtime/StaticClasspathResourceScanner.java#L57-L70

and

https://github.com/michael-simons/neo4j-migrations/blob/0db309476f9c737b1b4674193f838f7e3c74ec75/neo4j-migrations-quarkus-parent/runtime/src/main/java/ac/simons/neo4j/migrations/quarkus/runtime/StaticJavaBasedMigrationDiscoverer.java#L48-L61

are invoked from a build time processor, both in 2 steps (creating the scanner, than executing them and adding things to the augmented classes), like this:

https://github.com/michael-simons/neo4j-migrations/blob/0db309476f9c737b1b4674193f838f7e3c74ec75/neo4j-migrations-quarkus-parent/deployment/src/main/java/ac/simons/neo4j/migrations/quarkus/deployment/MigrationsProcessor.java#L58-L83

With Quarkus 2.7.x this suddenly fails: https://github.com/michael-simons/neo4j-migrations/runs/5002711436?check_suite_focus=true#step:9:1522 no resources are found anymore.

What fails exactly is

try (var scanResult = new ClassGraph().acceptPaths(paths).scan()) {
			var resources = scanResult.getResourcesWithExtension(Defaults.CYPHER_SCRIPT_EXTENSION).stream()
}

and the counterpart for Java classes.

It doesn't work when I use the current thread class loader either.

It works during runtime, but this is not what I want.

So, I'm unsure where to open an issue: In ClassGraph or in Quarkus.
I have tendency towards Quarkus though, as ClassGraph hasn't change, but Quarkus seems to have.

The thing is reproducible with the above project in this branch https://github.com/michael-simons/neo4j-migrations/tree/dependabot/maven%2Fquarkus.version-2.7.0.Final

when building like this:

./mvnw -am -pl :neo4j-migrations-quarkus-integration-tests -pl :neo4j-migrations-quarkus -pl :neo4j-migrations-quarkus-deployment clean verify

(Requires JDK 17).

Any pointers would be much appreciated, I'd like to celebrate Quarkus 2.7 with you :)

[aloubyansky]

Please, don't use the ClassGraph API in Quarkus. It's based on assumptions about internal implementation of the QuarkusClassLoader and may break any time, which is what happened in your case.
Thread.currentThread().getContextClassLoader() should work for you though.

[michael-simons]

Hmm… That's not helpful.

a) I tried passing Thread.currentThread().getContextClassLoader() to class graph (using the override class loader setup and ignoring parent class loader): Still no results
b) My project is build around using that API. BUT: I could remove the need for in the Quarkus extension, but what kind of alternatives does Quarkus offer for

• finding resources
• finding classes implementing a given interface

Thanks.

[aloubyansky]

What do you mean by ignoring parent class loader though?

[michael-simons]

Doesn't matter if you don't bother trying to understand what's the point of using class graph :(

[aloubyansky]

I do get the point of it. I am just saying it's currently broken, unfortunately.

[aloubyansky]

As I mentioned previously, you should be able to load classpath resources using Thread.currentThread().getContextClassLoader(). If it does not find them they are not on the classpath.

[michael-simons]

But that's the whole point of using the Class Graph API :( Finding the classes (and resources).

I hoped I made the purpose of the project clear: It pretty much does what Flyway does, but for graphs. It needs to be able to discover resources and classes during build. It worked on Quarkus 2.6.x.

[aloubyansky]

During augmentation there is a classloader hierarchy and you should not filter out parent classloaders.

[aloubyansky]

Sorry, then they should keep up with the changes in Quarkus.

[michael-simons]

So I'm gonna create an issue with Classgraph, then? (Which is not my project, just to be clear).

But yes, that was the original question…

[aloubyansky]

You can do that, sure. Or eliminate the need for dependency on ClassGraph in your code.

[michael-simons]

That discussion makes me not happy.

Anyhow, thanks.

[aloubyansky]

Sorry, but have a look at how ClassGraph implemented support for QuarkusClassLoader. It's very fragile. We can't guarantee it's going to work. I could help finding an alternative relying on a more stable API.

[aloubyansky]

You could also look into Jandex. This is what Quarkus uses internally to scan classpath.

[michael-simons]

Is there any documentation near as good as ClassGraph?

[aloubyansky]

Probably not https://github.com/wildfly/jandex

[gsmet]

@aloubyansky would it be possible to provide a SPI to Classgraph that they could use? Because it's a pretty popular library and they made the effort to support Quarkus, even if it's with a hack, so IMHO we should be nice to them if we can.

[aloubyansky]

We could collaborate on that, sure. The more libraries are properly working with Quarkus the better. My answer was about the current state of things. I wouldn't recommend it today.

[michael-simons]

FYI classgraph/classgraph#642 instead of wasting energy debating here.

@gsmet it would be helpful to have your input here. ClassGraph is an important tool, and it isn't broken. As a matter of fact, it has been working with Quarkus just fine since 2020. Also, it's approach is not hacky, it's just not using dependencies to get access to the elements. I would see it as a compliment that someone contributed the handler to make it work on Quarkus (as it already does on Boot and a broad range of other containers).

From my perspective, an SPI somewhere on that level of that element interface would be helpful. Or at least some guarantees.

[lukehutch]

Apologies for the delay in responding to this, since I have been traveling...

@michael-simons contributed a fix for the Quarkus classloader changes to ClassGraph, and those fixes were merged and released as ClassGraph version 4.8.139.

As far as whether ClassGraph's approach is hacky: well, if a classloader does not expose its classpath through a supported public API, then there is literally no way of getting the classpath other than via reflection of non-public methods and fields. Of course reflection will fail eventually because of strong encapsulation, in fact reflection into the Quarkus classloader probably already fails today on JDK 16+, so you really need to work with the Quarkus creators to have them define a supported public API for getting the classpath.

For the record, I built a workaround to the encapsulation issue, called Narcissus, and this can be used as a stop-gap in the meanwhile until the classloader is fixed (Narcissus will be used automatically by ClassGraph, if it is present on the classpath, and if the platform is supported).

[aloubyansky]

Right. We haven't had this contract on the Quarkus side so far and it was just a matter of time when the current integration would break. Not blaming your efforts @lukehutch, you couldn't do better than that.
Regarding establishing a more reliable classpath API, not sure whether we should continue the discussion here or move somewhere else. There are a few things to consider:

1. Quarkus apps may run in different modes: prod, dev and test. The classloading model in dev and test isn't flat, i.e. there will be two Quarkus classloaders on top of the default one.
2. If a path on the classpath points to a directory, in some cases not all the content of that directory may be visible, i.e. there could be directory content filters. This is new in 2.7.0.Final. The new PathTreeClassPathElement added in 2.7.0.Final, which pretty much replaces the Directory- and JarClassPathElements, hides away the details whether the element is an archive or a directory and also could filter out some content. It's based on https://github.com/quarkusio/quarkus/blob/main/independent-projects/bootstrap/app-model/src/main/java/io/quarkus/paths/PathTree.java
3. There is also MemoryClassPathElement that represents classpath resources generated at runtime that might not be available as files.

If I understand it correctly, you would like to have a reliable way to visit all the resources? By resources I mean not just a list of JARs and directories but all the entries of the JARs and directories on the classpath.