Use ecj from maven #1

trancexpress · 2025-07-07T11:54:22Z

See original issue: salesforce#34

trancexpress · 2025-07-07T11:56:27Z

@guw here is the PR.

guw · 2025-07-07T12:26:12Z

The test indeed fails.

trancexpress · 2025-07-07T12:46:46Z

The test indeed fails.

I'll try to find time in the next days to fix the change.

guw · 2025-07-07T12:54:31Z

Thank you! You cannot print to System.out because Bazel uses that for communication with the worker process. Thus, you need to invoke the compile process manually and attach a debugger and step through.

See Debugging here:
https://github.com/eclipseguru/bazel-jdt-java-toolchain?tab=readme-ov-file#debugging

I also updated the Bazel Eclipse plug-in to allow working with the bzlmod toolchain repo. You can grab the latest at https://eclipseguru.github.io/bazel-eclipse/latest/

trancexpress · 2025-07-07T20:42:30Z

While preparing with the debugging steps, I'm not noticing a -Xecj_collect_used_deps=... line among the bazel arguments. @guw , just in case, is the parameter missing?

SUBCOMMAND: # //java:ExampleTest [action 'Building java/ExampleTest.jar (1 source file)', configuration: 87b914b02324b66eeb370d1d147add2173c14535c83f67951c94524601a5affa, execution platform: @@platforms//host:host, mnemonic: Javac]
(cd /workspaces/socbm775/sandreev/.bazel/06388c54d553ef56e97acc18353696b6/execroot/_main && \
  exec env - \
    LC_CTYPE=C.UTF-8 \
    PATH=...\
  external/rules_java++toolchains+remotejdk21_linux/bin/java \
    '--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED' \
    '--add-exports=jdk.compiler/com.sun.tools.javac.main=ALL-UNNAMED' \
    '--add-exports=jdk.compiler/com.sun.tools.javac.model=ALL-UNNAMED' \
    '--add-exports=jdk.compiler/com.sun.tools.javac.processing=ALL-UNNAMED' \
    '--add-exports=jdk.compiler/com.sun.tools.javac.resources=ALL-UNNAMED' \
    '--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED' \
    '--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED' \
    '--add-opens=jdk.compiler/com.sun.tools.javac.code=ALL-UNNAMED' \
    '--add-opens=jdk.compiler/com.sun.tools.javac.comp=ALL-UNNAMED' \
    '--add-opens=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED' \
    '--add-opens=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED' \
    '--add-opens=java.base/java.nio=ALL-UNNAMED' \
    '--add-opens=java.base/java.lang=ALL-UNNAMED' \
    '-Dsun.io.useCanonCaches=false' \
    -XX:-CompactStrings \
    -Xlog:disable \
    '-Xlog:all=warning:stderr:uptime,level,tags' \
    -jar \
    external/rules_jdt_java_toolchain+/compiler/export/JdtJavaBuilder_deploy.jar \
    @bazel-out/k8-fastbuild/bin/java/ExampleTest.jar-0.params \
    @bazel-out/k8-fastbuild/bin/java/ExampleTest.jar-1.params)
# Configuration: 87b914b02324b66eeb370d1d147add2173c14535c83f67951c94524601a5affa
# Execution platform: @@platforms//host:host
# Runner: worker
INFO: From Building java/ExampleTest.jar (1 source file):
[parsing    /home/sandreev/git/bazel/bazel-jdt-java-toolchain/examples/java/example/test/ExampleTest.java - #1/1]
[reading    java/lang/Object.class]
[analyzing  /home/sandreev/git/bazel/bazel-jdt-java-toolchain/examples/java/example/test/ExampleTest.java - #1/1]
[reading    org/junit/jupiter/api/Test.class]
[writing    example/test/ExampleTest.class - #1]
[completed  /home/sandreev/git/bazel/bazel-jdt-java-toolchain/examples/java/example/test/ExampleTest.java - #1/1]
[1 unit compiled]
[1 .class file generated]

$ cat ./execroot/_main/bazel-out/k8-fastbuild/bin/java/ExampleTest.jar-0.params
--output
bazel-out/k8-fastbuild/bin/java/ExampleTest.jar
--native_header_output
bazel-out/k8-fastbuild/bin/java/ExampleTest-native-header.jar
--output_manifest_proto
bazel-out/k8-fastbuild/bin/java/ExampleTest.jar_manifest_proto
--compress_jar
--output_deps_proto
bazel-out/k8-fastbuild/bin/java/ExampleTest.jdeps
--bootclasspath
bazel-out/k8-fastbuild/bin/external/rules_java+/toolchains/platformclasspath.jar
--system
external/rules_java++toolchains+remotejdk21_linux
--sources
java/example/test/ExampleTest.java
--javacopts
-source
21
-target
21
-warn:all
-warn:none
-verbose
--
--target_label
//java:ExampleTest
--strict_java_deps
ERROR
--direct_dependencies
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/platform/junit-platform-launcher/1.13.3/header_junit-platform-launcher-1.13.3.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/platform/junit-platform-reporting/1.13.3/header_junit-platform-reporting-1.13.3.jar
bazel-out/k8-fastbuild/bin/external/bazel_tools/tools/jdk/_ijar/TestRunner/rules_java++toolchains+remote_java_tools/java_tools/Runner_deploy-ijar.jar
--experimental_fix_deps_tool
add_dep

$ cat ./execroot/_main/bazel-out/k8-fastbuild/bin/java/ExampleTest.jar-1.params
--classpath
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/platform/junit-platform-launcher/1.13.3/header_junit-platform-launcher-1.13.3.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/platform/junit-platform-reporting/1.13.3/header_junit-platform-reporting-1.13.3.jar
bazel-out/k8-fastbuild/bin/external/bazel_tools/tools/jdk/_ijar/TestRunner/rules_java++toolchains+remote_java_tools/java_tools/Runner_deploy-ijar.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/apiguardian/apiguardian-api/1.1.2/header_apiguardian-api-1.1.2.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/platform/junit-platform-commons/1.13.3/header_junit-platform-commons-1.13.3.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/opentest4j/opentest4j/1.3.0/header_opentest4j-1.3.0.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/platform/junit-platform-engine/1.13.3/header_junit-platform-engine-1.13.3.jar
bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/opentest4j/reporting/open-test-reporting-tooling-spi/0.2.3/header_open-test-reporting-tooling-spi-0.2.3.jar
--reduce_classpath_mode
JAVABUILDER_REDUCED

Anyway I'll debug tomorrow, I do have the java -jar ... command line I need.

guw · 2025-07-08T04:55:35Z

It defaults to "all" when not specified.

bazel-jdt-java-toolchain/compiler/src/main/buildjar/com/google/devtools/build/buildjar/javac/BlazeEcjMain.java

Line 94 in 52b4334

return All; // default to all

trancexpress · 2025-07-08T06:39:28Z

Here is a URI of a binary type we visit:

jar:file:///./execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar!/org/junit/jupiter/api/Test.class

We turn this into:

/./execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar

And then fail to match this against:

./execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar

I'm not sure why there are 3 slashes instead of 2...

guw · 2025-07-08T07:11:46Z

Three slashes are ok for file URIs. It's file://<host>/<path> and <host> is meaningless. Thus, three slashes.

The more brittle part is the conversion back to relative paths. Bazel expects them all to be sandbox relative. We have this for detecting the base path. This should remove ./execroot/_main/ later.

bazel-jdt-java-toolchain/compiler/src/main/buildjar/com/google/devtools/build/buildjar/javac/BlazeEcjMain.java

Line 417 in 52b4334

    
           private static String detectWorkingDirPathPrefix(BlazeJavacArguments arguments) throws IOException {

guw · 2025-07-08T07:28:41Z

compiler/src/main/buildjar/com/google/devtools/build/buildjar/javac/BlazeEcjMain.java

+			if (answer.getBinaryType() != null) {
+				URI binaryTypeUri = answer.getBinaryType().getURI();
+				String uri= binaryTypeUri.toString();
+				String prefix = "jar:file://";


It might not hurt putting a comment in here, with a link to https://github.com/eclipse-jdt/eclipse.jdt.core/blob/87368ab46448493820f5bcba6d2442ecb25267c3/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/classfmt/ClassFileReader.java#L165 (to give context what the expected URI format should be)

trancexpress · 2025-07-08T08:11:25Z

detectWorkingDirPathPrefix

At least during the test (i.e. compiling the test class), user.dir is set to the same value as BlazeEcjMain.BlazeEclipseBatchCompiler.sandboxPathPrefix. If you mean we should use it to define the host part of the URI away, I don't see how; the code already tries to use sandboxPathPrefix for that. But since the URI states its an absolute path (/./execroot/_main/, the extra slash that I don't see the reason for), trying to make the jar path relative to sandboxPathPrefix results in some path that doesn't exist:

../../../../execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar

$ ls -la /.../.bazel/06388c54d553ef56e97acc18353696b6/../../../../execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar
ls: cannot access '/.../.bazel/06388c54d553ef56e97acc18353696b6/../../../../execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar': No such file or directory

Where the actual file is:

/.../.bazel/06388c54d553ef56e97acc18353696b6/execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar-r-xr-xr-x 1 sandreev users 240364 Jul  7 22:10 /localworkspaces/sandreev/.bazel/06388c54d553ef56e97acc18353696b6/execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar

I.e. the leading slash must be removed in some way, but I don't know if we can just do that (since I have no knowledge of what bazel is doing here, maybe its defining its own file system for the compile).

guw · 2025-07-08T08:25:54Z

I.e. the leading slash must be removed in some way, but I don't know if we can just do that (since I have no knowledge of what bazel is doing here, maybe its defining its own file system for the compile).

The handling of the slash is independent from Bazel. That's an URI/ECJ thing. The URI is constructed by ECJ. Thus, we need to apply the reverse logic to get back to the original path.

The URI is created here:
https://github.com/eclipse-jdt/eclipse.jdt.core/blob/87368ab46448493820f5bcba6d2442ecb25267c3/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/classfmt/ClassFileReader.java#L165

URI uri =  URI.create("jar:file://" + toUri(zip.getName()).getRawPath() + "!/" + filename);

the "extra" slash is added by ECJ here:
https://github.com/eclipse-jdt/eclipse.jdt.core/blob/87368ab46448493820f5bcba6d2442ecb25267c3/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/classfmt/ClassFileReader.java#L181-L183

String p = absoluteNormalFilePath.replace(File.separatorChar, '/');
if (!p.startsWith("/")) {
	p = "/" + p;
}

Bazel executes all actions within a sandbox. This is to ensure they only access declared inputs. The sandbox path is something internal. Normally this isn't a problem with most tools because all paths within the sandbox are relative from the working directory. However, ECJ converts all paths to absolute, canonical real paths. Thus we need to convert them back to relative paths for reporting. Otherwise IDEs (eg., IntelliJ) and other tools parsing the output wouldn't be able to match the jar locations.

This is working with ClasspathLocation.getPath today:

bazel-jdt-java-toolchain/compiler/src/main/buildjar/com/google/devtools/build/buildjar/javac/BlazeEcjMain.java

Lines 224 to 227 in a0014d5

    
           // we assume jars come from the execroot; JDT uses absolute/canonical paths 
        
           // therefore we translate the path back into an execroot relative path for Bazel 
        
           // to be happy 
        
           jarPath = sandboxPathPrefix.relativize(jarPath);

guw · 2025-07-08T08:32:28Z

From an implementation perspective, we might be able to ignore the extra slash and just attempt to find sandboxPathPrefix path (with trailing slash) within the uri string. If we do, we remove everything from the beginning till including sandboxPathPrefix from the path string and use the remaining substring as a match.

trancexpress · 2025-07-08T10:26:31Z

OK, I debugged also the old version of the code:

		protected void recordNameEnvironmentAnswer(ClasspathAnswer classpathAnswer) {
			Classpath classpath = classpathAnswer.source;
			if (classpath instanceof ClasspathLocation) {
				String jar = ((ClasspathLocation) classpath).getPath();
				if (jar != null && jar.endsWith(".jar")) {
					Path jarPath = Path.of(jar);
					if (processedJars.add(jarPath)) {
						// we assume jars come from the execroot; JDT uses absolute/canonical paths
						// therefore we translate the path back into an execroot relative path for Bazel
						// to be happy
						jarPath = sandboxPathPrefix.relativize(jarPath);

Before the relativize call, jarPath has this value:

/.../.bazel/06388c54d553ef56e97acc18353696b6/execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar

And after:

execroot/_main/bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar

guw · 2025-07-08T10:37:37Z

That is unexpected. The one in the produced jdeps file starts at bazel-out/..., which is expected because the working directory of the compilation process should be .../_bazel_username/hash/execroot/_main.

Example on my Mac:

cat bazel-bin/java/ExampleTest.jdeps

?
?bazel-out/darwin_arm64-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar//java:ExampleTest"
                                                                                                                                                                                example.test⏎

trancexpress · 2025-07-08T10:58:17Z

@guw does it make more sense for you to take a look? I don't know which values are expected and which are not. In addition I can't get bazel-bin/java/ExampleTest.jdeps to be created reliably:

cd /.../git/bazel/bazel-jdt-java-toolchain && bazelisk build //:JdtJavaBuilder_deploy.jar && cp -fv bazel-bin/JdtJavaBuilder_deploy.jar compiler/export/ && cd examples && bazelisk build --config=ecj --java_runtime_version=21  --java_language_version=21 //...

find .. -name ExampleTest.jdeps then returns nothing. Could of course be due to the changes in the PR... Though on the main branch I also can't get the file to show up reliably.

trancexpress · 2025-07-08T11:38:02Z

Maybe I'm invoking the debugging from the wrong folder? I do it in the root of the bazel folder that contains all the build stuff... Considering that sandboxPathPrefix is /.../sandreev/.bazel/92712ddfe14455ece296d843164c7ecf

I guess I can in the folder you mentioned... Then again there are 2:

./execroot/_main/bazel-out
./bazel-out

trancexpress · 2025-07-08T12:20:38Z

OK, when I cd to execroot/_main/ the URI is: jar:file:///bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar!/org/junit/jupiter/api/Test.class

What gets added to directDependenciesMap on the main branch:

path: "bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar"
kind: EXPLICIT

What gets added on my branch gh34:

path: "bazel-out/k8-fastbuild/bin/external/rules_jvm_external++maven+maven/org/junit/jupiter/junit-jupiter-api/5.13.3/header_junit-jupiter-api-5.13.3.jar"
kind: EXPLICIT

Still though, no idea what criteria we should use to remove the leading slash. Maybe its enough to just remove it, considering the check directJars.contains(jarPath)?

guw · 2025-07-08T13:04:56Z

Ah ok, yes. It's all relative to the working directory. If the logic adapts to the folder it's working.

guw · 2025-07-08T13:07:09Z

I'm comfortable with adjusting the expected (supported) prefix to jar:file:/// with a link to the source line in ECJ.

trancexpress · 2025-07-08T20:26:41Z

Thank you for merging this @guw ! We'll start using the repo now for our builds, instead of: https://github.com/salesforce/bazel-jdt-java-toolchain

guw · 2025-07-09T07:31:46Z

Absolutely. Please keep sending patched if you need additional features.

trancexpress · 2025-07-10T08:22:10Z

@guw the README is now outdated: https://github.com/eclipseguru/bazel-jdt-java-toolchain/blob/main/README.md

The section about compiling ecj in particular.

This one too maybe is outdated: https://github.com/eclipseguru/bazel-jdt-java-toolchain/blob/main/ecj-test/test-compile.sh

Do you have time to update it or should I try?

guw · 2025-07-10T12:24:05Z

Totally forgot about it. If you have time please don't hesitate!

trancexpress added 2 commits July 7, 2025 13:55

Delete ecj sources, as ecj will be downloaded from mvn

1059854

Download ecj from mvn

55faf58

trancexpress force-pushed the gh34 branch from 486f8da to 334ee90 Compare July 7, 2025 11:56

guw reviewed Jul 8, 2025

View reviewed changes

trancexpress force-pushed the gh34 branch from 334ee90 to 9c77707 Compare July 8, 2025 10:55

trancexpress force-pushed the gh34 branch from 9c77707 to 2713886 Compare July 8, 2025 12:17

Adjust BlazeEcjMain to changes in ecj

e753f8c

trancexpress force-pushed the gh34 branch from 2713886 to e753f8c Compare July 8, 2025 13:30

guw approved these changes Jul 8, 2025

View reviewed changes

guw enabled auto-merge (squash) July 8, 2025 13:59

Merge branch 'main' into gh34

b190211

guw merged commit 2f50834 into eclipseguru:main Jul 8, 2025
22 checks passed

Use ecj from maven #1

Use ecj from maven #1

Uh oh!

Conversation

trancexpress commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trancexpress commented Jul 7, 2025

Uh oh!

guw commented Jul 7, 2025

Uh oh!

trancexpress commented Jul 7, 2025

Uh oh!

guw commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trancexpress commented Jul 7, 2025

Uh oh!

guw commented Jul 8, 2025

Uh oh!

trancexpress commented Jul 8, 2025

Uh oh!

guw commented Jul 8, 2025

Uh oh!

guw Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

trancexpress commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guw commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guw commented Jul 8, 2025

Uh oh!

trancexpress commented Jul 8, 2025

Uh oh!

guw commented Jul 8, 2025

Uh oh!

trancexpress commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trancexpress commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trancexpress commented Jul 8, 2025

Uh oh!

guw commented Jul 8, 2025

Uh oh!

guw commented Jul 8, 2025

Uh oh!

Uh oh!

trancexpress commented Jul 8, 2025

Uh oh!

guw commented Jul 9, 2025

Uh oh!

trancexpress commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guw commented Jul 10, 2025

Uh oh!

Uh oh!

trancexpress commented Jul 7, 2025 •

edited

Loading

guw commented Jul 7, 2025 •

edited

Loading

trancexpress commented Jul 8, 2025 •

edited

Loading

guw commented Jul 8, 2025 •

edited

Loading

trancexpress commented Jul 8, 2025 •

edited

Loading

trancexpress commented Jul 8, 2025 •

edited

Loading

trancexpress commented Jul 10, 2025 •

edited

Loading