Unable to find mapped JDK in AdoptOpenJDK Docker images after mid-October

[sophia-guo]

Starting in mid October (around Oct 14th), we see issues running java executable from a mapped drive using the 'nightly' images produced at the project and pushed to Docker hub. This was previously not an issue.

Suspect that something changed in the Docker images as test material and mapped JDK is unchanged when issue is present (so this may get shifted to openjdk-docker repo after additional investigation).

Running Tomee tests using the JDK from within the AdoptOpenJDK Docker image,
https://ci.adoptopenjdk.net/job/Grinder/4315/console you can see that the tests execute and run:

09:19:13  	/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../external/external.sh --clean --tag nightly --version 8 --impl openj9 --dir tomee; } 2>&1 | tee -a "/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16037183519158/TestTargetResult";
09:19:13  docker run --rm  --rm adoptopenjdk-tomee-test:8-jdk-ubuntu-openj9-full 
09:19:13  Using docker image default Java
09:19:13  JAVA_BIN is: /opt/java/openjdk/bin
09:19:13  APPLICATION_NAME=tomee
09:19:13  APPLICATION_TAG=tomee-8.0.2
09:19:13  OS_TAG=ubuntu
09:19:13  =JAVA VERSION OUTPUT BEGIN=
09:19:13  openjdk version "1.8.0_272"
09:19:13  OpenJDK Runtime Environment (build 1.8.0_272-202010231739-b10)
09:19:13  Eclipse OpenJ9 VM (build master-67af03f58, JRE 1.8.0 Linux s390x-64-Bit Compressed References 20201023_805 (JIT enabled, AOT enabled)
09:19:13  OpenJ9   - 67af03f58
09:19:13  OMR      - cb68c30b7
09:19:13  JCL      - bea7d86207 based on jdk8u272-b10)
09:19:13  =JAVA VERSION OUTPUT END=
09:19:13  Build TomEE without running test
09:19:14  [INFO] Scanning for projects...
09:19:14  [INFO] Downloading from apache: https://repository.apache.org/content/repositories/releases/org/apache/apache/21/apache-21.pom
...
...
10:02:12  [INFO] ------------------------------------------------------------------------
10:02:12  [INFO] Reactor Summary for TomEE :: TCK :: MicroProfile 8.0.2:
10:02:12  [INFO] 
10:02:12  [INFO] TomEE :: TCK :: MicroProfile ....................... SUCCESS [  0.375 s]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Config TCK ............ SUCCESS [01:38 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile JWT TCK ............... SUCCESS [01:17 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Fault Tolerance TCK ... SUCCESS [08:59 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Health TCK ............ SUCCESS [ 49.368 s]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Metrics TCK ........... SUCCESS [06:13 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Rest Client TCK ....... SUCCESS [06:54 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Open API TCK .......... SUCCESS [01:56 min]
10:02:12  [INFO] TomEE :: TCK :: MicroProfile Open Tracing TCK ...... SUCCESS [ 40.002 s]
10:02:12  [INFO] ------------------------------------------------------------------------
10:02:12  [INFO] BUILD SUCCESS
10:02:12  [INFO] ------------------------------------------------------------------------

Running same Tomee tests with mapped drive to nightly JDK build, Grinder 4314, unable to determine mapped JDK (JAVA VERSION OUTPUT is empty).

09:18:46  	/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../external/external.sh --clean --tag nightly --version 8 --impl openj9 --dir tomee; } 2>&1 | tee -a "/home/jenkins/workspace/Grinder/openjdk-tests/TKG/../TKG/test_output_16037183262603/TestTargetResult";
09:18:46  docker run --rm -v /home/jenkins/workspace/Grinder/openjdkbinary/j2sdk-image:/opt/java/openjdk --rm adoptopenjdk-tomee-test:8-jdk-ubuntu-openj9-full 
09:18:47  Using docker image default Java
09:18:47  JAVA_BIN is: /opt/java/openjdk/bin
09:18:47  APPLICATION_NAME=tomee
09:18:47  APPLICATION_TAG=tomee-8.0.2
09:18:47  OS_TAG=ubuntu
09:18:47  =JAVA VERSION OUTPUT BEGIN=
09:18:47  =JAVA VERSION OUTPUT END=
09:18:47  Build TomEE without running test
09:18:47  
09:18:47  tomee_test_j9_0_FAILED
09:18:47  

Details in adoptium/aqa-tests#2031

adoptium/aqa-tests#2031 (comment)

[bharathappali]

Exploration done till now (Adding here for quick reference):
adoptium/aqa-tests#2031 (comment)
adoptium/aqa-tests#2031 (comment)

[bharathappali]

@sophia-guo I have raised an issue in OpenJ9 eclipse-openj9/openj9#11110

I have pointed your issue in adoptium/aqa-tests#2031 as it has more details than this issue in openjdk-docker repo.

[bharathappali]

From the conversation at OpenJ9 issue eclipse-openj9/openj9#11110 (comment) , It looks like the current SCC mechanism is not handling a mismatch wrt SCC generated by different builds as expected. A quick possible workaround would be to repopulate the cache by removing readOnly when we mount the jdk and run the tests.

so while running docker we add a env var OPENJ9_JAVA_OPTIONS and set it to -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,nonfatal

Eg :

docker run --rm -e OPENJ9_JAVA_OPTIONS=-Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,nonfatal -v /Users/sophiag/Downloads/dockermount/jdk8u272-b10_release:/opt/java/openjdk -it adoptopenjdk/openjdk8-openj9:nightly java -version

So that the cache is re-created, JVM starts up without any error (related to SCC) and tests can be performed.

@dinogun @smlambert @sophia-guo Can i have your views on this workaround ?

[bharathappali]

eclipse-openj9/openj9#11213 fixes the issue of JVM crashing on nightly build SCC.

I hope the next release of OpenJ9 will have the changes to fix this problem.

EDIT:

FYI - eclipse-openj9/openj9#11213 (comment)