Two jdk8 security tests consistently failed on test-osuosl-centos74-ppc64le-1

[sophia-guo]

Recent builds run shows that JDK8 security tests:
sun/tools/jinfo/Basic.sh
sun/security/pkcs11/fips/TestTLS12.java

consistently failed on test-osuosl-centos74-ppc64le-1.

• test suite/name? sun/tools/jinfo/Basic.sh sun/security/pkcs11/fips/TestTLS12.java
• Is there an existing issue elsewhere covering this?
• Which machine(s) does it work on? test-osuosl-ubuntu1804-ppc64le-1, test-osuosl-ubuntu1804-ppc64le-2, test-osuosl-ubuntu2004-ppc64le-1
• Which machine(s) does it fail on? test-osuosl-centos74-ppc64le-1
• Do you have a link to a Grinder re-run if the test with the failure? https://ci.adoptopenjdk.net/job/Grinder/parambuild/?SDK_RESOURCE=nightly&TARGET=jdk_custom&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=-v+%24%7BTEST_JDK_HOME%7D%3A%2Fopt%2Fjava%2Fopenjdk&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&OPENJ9_SYSTEMTEST_OWNER_BRANCH=eclipse-openj9%3Amaster&PLATFORM=ppc64le_linux&GENERATE_JOBS=false&KEEP_REPORTDIR=true&PERSONAL_BUILD=true&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2Fadoptium%2Faqa-tests.git&LABEL=test-osuosl-centos74-ppc64le-1&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=8&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&UPSTREAM_JOB_NAME=&CLOUD_PROVIDER=&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=sun%2Fsecurity%2Fpkcs11%2Ffips%2FTestTLS12.java&VENDOR_TEST_SHAS=&JDK_BRANCH=&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_IMPL=hotspot&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&DYNAMIC_COMPILE=false&ADOPTOPENJDK_SYSTEMTEST_OWNER_BRANCH=adoptium%3Amaster&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=openjdk&USE_TESTENV_PROPERTIES=false&UPSTREAM_JOB_NUMBER=&STF_OWNER_BRANCH=adoptium%3Amaster&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None

Any other details:
Grinder on test-osuosl-ubuntu1804-ppc64le-1: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5017/, https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5016

Grinder on test-osuosl-centos74-ppc64le-1: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5015, https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5014

[Haroon-Khel]

Rerunning on test-osuosl-centos74-ppc64le-1

sun/security/pkcs11/fips/TestTLS12.java
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5092/console

sun/tools/jinfo/Basic.sh
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5094/console

[Haroon-Khel]

Other ppc64 linux machines

sun/security/pkcs11/fips/TestTLS12.java

test-osuosl-centos74-ppc64le-2
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5095/console

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5096/console
test-osuosl-ubuntu1804-ppc64le-2

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5097/console
test-skytap-ubuntu2004-ppc64le-1

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5098/console
test-osuosl-ubuntu1804-ppc64le-1

sun/tools/jinfo/Basic.sh

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5100/console
test-skytap-ubuntu2004-ppc64le-1

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5101/console
test-osuosl-ubuntu1804-ppc64le-2

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5102/console
test-osuosl-ubuntu1804-ppc64le-1

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5103/console
test-osuosl-centos74-ppc64le-2

[Haroon-Khel]

Both tests also fail on test-osuosl-centos74-ppc64le-2

[Haroon-Khel]

For the sun/tools/jinfo/Basic.sh failure, the following line causes the test to fail
https://github.com/adoptium/jdk8u/blob/a18e9043fa2a0a14098e1ec25d32577aaac6c023/jdk/test/sun/tools/jinfo/Basic.sh#L64

Edit: The test failure is caused by that if statement block (lines 62-75) in the linked file.

Using jinfo with either the -sysprops, -flags options, or no option, will cause the error. Meanwhile using the -flag option (the later tests in the linked file) will not cause the error

[Haroon-Khel]

Testing this in a command line environment on a failing machine, test-osuosl-centos74-ppc64le-1:

[jenkins@test-centos74-1 bin]$ ./jinfo -J-XX:+UsePerfData -flags +PrintGC 21443
Usage:
    jinfo <option> <pid>
       (to connect to a running process)

where <option> is one of:
    -flag <name>         to print the value of the named VM flag
    -flag [+|-]<name>    to enable or disable the named VM flag
    -flag <name>=<value> to set the named VM flag to the given value
    -h | -help           to print this help message
[jenkins@test-centos74-1 bin]$ ./jinfo -J-XX:+UsePerfData -sysprops 21443
Usage:
    jinfo <option> <pid>
       (to connect to a running process)

where <option> is one of:
    -flag <name>         to print the value of the named VM flag
    -flag [+|-]<name>    to enable or disable the named VM flag
    -flag <name>=<value> to set the named VM flag to the given value
    -h | -help           to print this help message
[jenkins@test-centos74-1 bin]$ ./jinfo -J-XX:+UsePerfData 21443
Usage:
    jinfo <option> <pid>
       (to connect to a running process)

where <option> is one of:
    -flag <name>         to print the value of the named VM flag
    -flag [+|-]<name>    to enable or disable the named VM flag
    -flag <name>=<value> to set the named VM flag to the given value
    -h | -help           to print this help message

While If I use one of the working options:

[jenkins@test-centos74-1 bin]$ ./jinfo -J-XX:+UsePerfData -flag -PrintGC 21443
[jenkins@test-centos74-1 bin]$ echo $?
0

[Haroon-Khel]

I think I've got it. These failing tests are not supposed to run on this machine. For example, on a passing machine, test-skytap-ubuntu2004-ppc64le-1, the if statement block which runs these failing tests does not run, https://github.com/adoptium/jdk8u/blob/a18e9043fa2a0a14098e1ec25d32577aaac6c023/jdk/test/sun/tools/jinfo/Basic.sh#L62, since the $runSA = true condition is false.

On test-skytap-ubuntu2004-ppc64le-1, the command ptrace_scope=/sbin/sysctl -n kernel.yama.ptrace_scope, from https://github.com/adoptium/jdk8u/blob/a18e9043fa2a0a14098e1ec25d32577aaac6c023/jdk/test/sun/tools/jinfo/Basic.sh#L54 returns 1 while on a failing machine, test-osuosl-centos74-ppc64le-1, it returns 0

[Haroon-Khel]

Setting kernel.yama.ptrace_scope to 1 has fixed this

[jenkins@test-centos74-1 bin]$ /sbin/sysctl -n kernel.yama.ptrace_scope
1
[jenkins@test-centos74-1 bin]$ ./java -jar ../../jtreg/lib/jtreg.jar ../../jdk8u/jdk/test/sun/tools/jinfo/Basic.sh
Test results: passed: 1
Report written to /home/jenkins/jdk8u332-b09/bin/JTreport/html/report.html
Results written to /home/jenkins/jdk8u332-b09/bin/JTwork

[Haroon-Khel]

Rerunning sun/tools/jinfo/Basic.sh

centos-1
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5170/console
centos-2
https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5171/console

[Haroon-Khel]

Ref sun/security/pkcs11/fips/TestTLS12.java

On a passing machine, test-skytap-ubuntu2004-ppc64le-1, this test is skipped due to
Test skipped: TLS 1.2 mechanisms not supported by current SunPKCS11 back-end

Need to find why it isnt skipped on the failing machine, or whether it should be skipped

[Haroon-Khel]

On a passing machine, the value for sunPKCS11NSSProvider is null, causing the test to skip. While on our 2 failing machines, this value is SunPKCS11-NSSKeyStore version 1.8