Hash of java.rmi differs from expected hash

[tresf]

Please provide a brief summary of the bug

Builds on GitHub Actions using GitHub's Temurin 11.0.16+8 against Temurin 11.0.12+7 are failing.

Please provide steps to reproduce where possible

I haven't yet reproduced this locally, but at this point all GitHub builds using jlink against the above combinations of Java 11 seem to be affected (MacOS, Linux, Linux targeting Windows)

• GitHub uses JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jdk/11.0.16-8/x64
• Project uses Temurin URL: https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.12+7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.12_7.tar.gz

Expected Results

jlink succeeds

Actual Results

jlink fails

What Java Version are you using?

JAVA_HOME: /usr/lib/jvm/temurin-11-jdk-amd64

What is your operating system and platform?

https://github.com/qzind/tray/runs/7662042575?check_suite_focus=true

Run actions/setup-java@v2
  with:
    java-version: 11
    distribution: temurin
    java-package: jdk
    architecture: x64
    check-latest: false
    server-id: github
    server-username: GITHUB_ACTOR
    server-password: GITHUB_TOKEN
    overwrite-settings: true
    job-status: success

How did you install Java?

GitHub actions/setup-java@v2

Did it work before?

Worked with 11.0.15+10

Did you test with other Java versions?

No response

Relevant log output

[java] [DEBUG] 2022-08-03T22:30:24,187 @ qz.utils.ShellUtilities:-1
     [java] 	Executing: [/usr/lib/jvm/temurin-11-jdk-amd64/bin/jlink, --strip-debug, --compress=2, --no-header-files, --no-man-pages, --exclude-files=glob:**/legal/**, --module-path, /home/runner/work/tray/tray/out/jlink/jdk-adoptopenjdk-x64-linux-11.0.12_7/jdk-11.0.12+7/jmods, --add-modules, java.base,java.compiler,java.datatransfer,java.desktop,java.logging,java.management,java.naming,java.net.http,java.prefs,java.rmi,java.scripting,java.security.jgss,java.sql,java.xml,jdk.jfr,jdk.jsobject,jdk.unsupported,jdk.unsupported.desktop,jdk.xml.dom,jdk.zipfs,jdk.crypto.ec, --output, /home/runner/work/tray/tray/out/dist/runtime]
     [java] Error: Hash of java.rmi (f9e89c6c0897e9f27f45f1299ad648dc5a4484c8a197ba19beb2f07900ff3f91) differs to expected hash (fd9918170a44656670ff2dcc5affa75e6a42f1e521c8c62145d27a4189108226) recorded in java.base
Error: ava] [ERROR] 2022-08-03T22:30:24,761 @ qz.utils.ArgParser:-1
     [java] 	Build step JLINK failed
     [java] java.io.IOException: An error occurred deploying the jre.  Please check the logs for details.

[tresf]

Similarly described issues:

• jlink error Hash of java.rmi differs to expected hash .. corretto/corretto-11#144 (comment)
• aarch64 build failure, Hash of java.rmi (...) differs to expected hash  temurin-build#1804

[tellison]

Doesn't look like a Github action issue.

I suggest checking that other vendors are not also seeing a difference in the jlink mod hash code of java.rmi between these dot releases, and that the problem is not an inadvertent change to the values used when calculating the mods hashes. Either that or we are running jlink in an incorrect way in the latest build method.

[tresf]

I suggest checking that other vendors are not also seeing a difference in the jlink mod hash code of java.rmi between these dot releases

The same issue was reported against the same version of Corretto here: corretto/corretto-11#144 (comment) but I don't really understand the message or how a user is intended to act upon this error. From what I can tell, jlink is unusable in this state unless there's a workaround.

How would one go about auditing other vendors? I'm not even sure what this message would represent. I assume it's some type of major version check, but if that's the case, why aren't these being picked up in unit tests? (Or perhaps they are but they haven't been backported to JKD11?)

[tresf]

Another proposal is to ask setup-java to use a locked-in version, but that must be weighed against losing the ability to know and thus report these issues.

[tresf]

In the meantime I'll try to create a unit test to reproduce however due to the history of this issue happening prior, I was hoping there'd be enough knowledge around it to take immediate action.

[tellison]

On my phone at the moment, so apologies for the terse replies. Yes, the Corretto (future) fix notice is here

https://github.com/corretto/corretto-11/blob/release-11.0.9.11.2/CHANGELOG.md#corretto-version-1109112

Looks like no code change was required.

[tresf]

I've asked them for the information about the patch; it seems quite coincidental to affect identical versions.

[lutkerd]

Hi, I'm on the Corretto team. We root caused it to openjdk/jdk11u@7c29f0e which changes how the hashes are calculated. We were using an older version of Corretto11 to repack the jmods after signing the binaries. No code changes were required for us to fix since we just rebuilt using 11.0.16 to repack.

This means you cannot use jlink across minor versions after 11.0.16.

[tresf]

This means you cannot use jlink across minor versions after 11.0.16.

Whoah, this breaks jlink at its core. Is this by design? Can you share some upstream conversations about this? jlink already has a high learning curve, I fear this is going to make adoption even harder.

[lutkerd]

I just checked and there was no conversation about it on the email list, JBS, or PR.
https://mail.openjdk.org/pipermail/jdk-updates-dev/2022-March/013027.html
https://bugs.openjdk.org/browse/JDK-8243666
openjdk/jdk11u-dev#915

[alvdavi]

The good news is that now that jlink is aligned with how JDK15+ perform the hash calculation, there should no longer be incompatibilities when repackaging JDK11 jmods using newer JDKs (which is what cause the original Corretto issue two years ago). After the lessons learned with this patch, we don't think changes to the jlink format should be done for existing major JDK versions, and even for new major versions should be avoided.

[tresf]

there should no longer be incompatibilities when repackaging JDK11 jmods using newer JDKs (which is what cause the original Corretto issue two years ago). After the lessons learned with this patch, we don't think changes to the jlink format should be done for existing major JDK versions

Hmm... this is very interesting... So does that mean this (following) statement is only true for =< 11.0.15? That might not be as dire of a consequence.

This means you cannot use jlink across minor versions after 11.0.16.

For example, in my builds, I check the major version only currently, but I can add an exception to 11.0.16+.

[tresf]

So does that mean this (following) statement is only true for =< 11.0.15? (that 11.0.16 will be forward-compatible with 11.0.17, etc?)

While I await an answer, I've written a patch in my environment that assumes 11.0.16 and higher can be used to build >=11.0.16 and that 11.0.15 and lower can be used to build <= 11.0.15 and it's succeeding.

I'm sure this can be simplified to just check if it's greater than 11.0.16, but speculative fear will keep the old logic around for a while. 😅

Once I have a definitive answer, we can close this as wontfix. 🍻

[lutkerd]

Our understanding is that all versions prior to 11.0.16 should work against each other and 11.0.16+ should work. I think we would push back on any more breaking changes here. 😄

[tresf]

Thanks closing as upstream has permanently broken backwards compatibility and a manual workaround is needed.

Quoting:

I just checked and there was no conversation about it on the email list, JBS, or PR.
https://mail.openjdk.org/pipermail/jdk-updates-dev/2022-March/013027.html
https://bugs.openjdk.org/browse/JDK-8243666
openjdk/jdk11u-dev#915

[keithc-ca]

Hashes of non-upgradeable modules are recorded in java.base.jmod (from the same version). If you try to mix java.base from one version and any of those non-upgradeable modules from another version it should be expected to fail in this way. I believe that is by design.

[tresf]

Hashes of non-upgradeable modules are recorded in java.base.jmod (from the same version). If you try to mix java.base from one version and any of those non-upgradeable modules from another version it should be expected to fail in this way. I believe that is by design.

Strongly disagree. According to this comment, the hash calculation was changed to be more reliable, but in doing so, it broke user space: #557 (comment)

Frameworks should be careful to not break user space, it adds unintuitive errors, breaks the documentation for such features, and deters people from trusting the feature.