[CI] IndexUpgradeIT Failures Related to Licensing

[jdconrad]

On 6.x the following 5 tests failed:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+release-tests/722/console

IndexUpgradeIT.testIndexUpgradeInfo -- fails consistently

Reproduce:

./gradlew :x-pack:plugin:upgrade:internalClusterTest   -Dtests.seed=55726863F3EDDD7D   -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT   -Dtests.method="testIndexUpgradeInfo"   -Dtests.security.manager=true   -Dbuild.snapshot=false   -Dtests.jvm.argline="-Dbuild.snapshot=false"   -Dtests.locale=sr-ME  -Dlicense.key=<your license key location>

Stack:

REPRODUCE WITH: ./gradlew :x-pack:plugin:upgrade:internalClusterTest -Dtests.seed=55726863F3EDDD7D -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT -Dtests.method="testIndexUpgradeInfo" -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=sr-ME -Dtests.timezone=Asia/Saigon
ERROR   2.67s | IndexUpgradeIT.testIndexUpgradeInfo <<< FAILURES!
   > Throwable #1: java.lang.IllegalStateException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException: DerInputStream.getLength(): lengthTag=14, too big.
   >    at __randomizedtesting.SeedInfo.seed([55726863F3EDDD7D:7ED3540B1839B7FE]:0)
   >    at org.elasticsearch.license.CryptUtils.readPublicKey(CryptUtils.java:84)
   >    at org.elasticsearch.license.licensor.LicenseSigner.sign(LicenseSigner.java:85)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:285)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:243)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:235)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.updateLicensing(IndexUpgradeIntegTestCase.java:61)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.enableLicensing(IndexUpgradeIntegTestCase.java:53)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIT.resetLicensing(IndexUpgradeIT.java:37)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)

IndexUpgradeIT.testIndexUpgradeInfoLicense - fails consistently

Reproduce:

./gradlew :x-pack:plugin:upgrade:internalClusterTest   -Dtests.seed=55726863F3EDDD7D   -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT   -Dtests.method="testIndexUpgradeInfoLicense"   -Dtests.security.manager=true   -Dbuild.snapshot=false   -Dtests.jvm.argline="-Dbuild.snapshot=false"   -Dtests.locale=sr-ME   -Dtests.timezone=Asia/Saigon   -Dlicense.key=<your license key location>

Stack:

  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:upgrade:internalClusterTest -Dtests.seed=55726863F3EDDD7D -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT -Dtests.method="testIndexUpgradeInfoLicense" -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=sr-ME -Dtests.timezone=Asia/Saigon
ERROR   2.58s | IndexUpgradeIT.testIndexUpgradeInfoLicense <<< FAILURES!
   > Throwable #1: java.lang.IllegalStateException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException: DerInputStream.getLength(): lengthTag=14, too big.
   >    at __randomizedtesting.SeedInfo.seed([55726863F3EDDD7D:842130AD685DB464]:0)
   >    at org.elasticsearch.license.CryptUtils.readPublicKey(CryptUtils.java:84)
   >    at org.elasticsearch.license.licensor.LicenseSigner.sign(LicenseSigner.java:85)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:285)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:243)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:235)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.updateLicensing(IndexUpgradeIntegTestCase.java:61)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.enableLicensing(IndexUpgradeIntegTestCase.java:53)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIT.resetLicensing(IndexUpgradeIT.java:37)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)

IndexUpgradeIT.testIndexUpgradeInfoOnEmptyCluster - fails consistently

Reproduce:

./gradlew :x-pack:plugin:upgrade:internalClusterTest   -Dtests.seed=55726863F3EDDD7D   -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT   -Dtests.method="testUpToDateIndexUpgrade"   -Dtests.security.manager=true   -Dbuild.snapshot=false   -Dtests.jvm.argline="-Dbuild.snapshot=false"   -Dtests.locale=sr-ME   -Dtests.timezone=Asia/Saigon   -Dlicense.key=<your public key location>

Stack:

  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:upgrade:internalClusterTest -Dtests.seed=55726863F3EDDD7D -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT -Dtests.method="testUpToDateIndexUpgrade" -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=sr-ME -Dtests.timezone=Asia/Saigon
ERROR   2.62s | IndexUpgradeIT.testUpToDateIndexUpgrade <<< FAILURES!
   > Throwable #1: java.lang.IllegalStateException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException: DerInputStream.getLength(): lengthTag=14, too big.
   >    at __randomizedtesting.SeedInfo.seed([55726863F3EDDD7D:6B91CA0F87E4F09E]:0)
   >    at org.elasticsearch.license.CryptUtils.readPublicKey(CryptUtils.java:84)
   >    at org.elasticsearch.license.licensor.LicenseSigner.sign(LicenseSigner.java:85)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:285)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:243)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:235)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.updateLicensing(IndexUpgradeIntegTestCase.java:61)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.enableLicensing(IndexUpgradeIntegTestCase.java:53)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIT.resetLicensing(IndexUpgradeIT.java:37)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)

IndexUpgradeIT.testInternalUpgradePrePostChecks - fails consistently

Reproduce:

./gradlew :x-pack:plugin:upgrade:internalClusterTest   -Dtests.seed=55726863F3EDDD7D   -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT   -Dtests.method="testIndexUpgradeInfoOnEmptyCluster"   -Dtests.security.manager=true   -Dbuild.snapshot=false   -Dtests.jvm.argline="-Dbuild.snapshot=false"   -Dtests.locale=sr-ME   -Dtests.timezone=Asia/Saigon   -Dlicense.key=<your public key here>

Stack:

  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:upgrade:internalClusterTest -Dtests.seed=55726863F3EDDD7D -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT -Dtests.method="testIndexUpgradeInfoOnEmptyCluster" -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=sr-ME -Dtests.timezone=Asia/Saigon
ERROR   2.55s | IndexUpgradeIT.testIndexUpgradeInfoOnEmptyCluster <<< FAILURES!
   > Throwable #1: java.lang.IllegalStateException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException: DerInputStream.getLength(): lengthTag=14, too big.
   >    at __randomizedtesting.SeedInfo.seed([55726863F3EDDD7D:7800CCD8EFFBE2AD]:0)
   >    at org.elasticsearch.license.CryptUtils.readPublicKey(CryptUtils.java:84)
   >    at org.elasticsearch.license.licensor.LicenseSigner.sign(LicenseSigner.java:85)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:285)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:243)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:235)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.updateLicensing(IndexUpgradeIntegTestCase.java:61)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.enableLicensing(IndexUpgradeIntegTestCase.java:53)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIT.resetLicensing(IndexUpgradeIT.java:37)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)

IndexUpgradeIT.testUpToDateIndexUpgrade - fails consistently

Reproduce:

./gradlew :x-pack:plugin:upgrade:internalClusterTest   -Dtests.seed=55726863F3EDDD7D   -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT   -Dtests.method="testInternalUpgradePrePostChecks"   -Dtests.security.manager=true   -Dbuild.snapshot=false   -Dtests.jvm.argline="-Dbuild.snapshot=false"   -Dtests.locale=sr-ME   -Dtests.timezone=Asia/Saigon   -Dlicense.key=<your public key here>

Stack:

  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:upgrade:internalClusterTest -Dtests.seed=55726863F3EDDD7D -Dtests.class=org.elasticsearch.xpack.upgrade.IndexUpgradeIT -Dtests.method="testInternalUpgradePrePostChecks" -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=sr-ME -Dtests.timezone=Asia/Saigon
ERROR   2.76s | IndexUpgradeIT.testInternalUpgradePrePostChecks <<< FAILURES!
   > Throwable #1: java.lang.IllegalStateException: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException: DerInputStream.getLength(): lengthTag=14, too big.
   >    at __randomizedtesting.SeedInfo.seed([55726863F3EDDD7D:BE77FBC296B86076]:0)
   >    at org.elasticsearch.license.CryptUtils.readPublicKey(CryptUtils.java:84)
   >    at org.elasticsearch.license.licensor.LicenseSigner.sign(LicenseSigner.java:85)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:285)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:243)
   >    at org.elasticsearch.license.TestUtils.generateSignedLicense(TestUtils.java:235)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.updateLicensing(IndexUpgradeIntegTestCase.java:61)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIntegTestCase.enableLicensing(IndexUpgradeIntegTestCase.java:53)
   >    at org.elasticsearch.xpack.upgrade.IndexUpgradeIT.resetLicensing(IndexUpgradeIT.java:37)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >    at java.base/java.lang.Thread.run(Thread.java:844)

[elasticmachine]

Pinging @elastic/es-core-infra

[jdconrad]

@jkakavas Hey Ioannis, please feel free to re-assign this as appropriate, but I thought this might be related to some recent FIPS licensing changes you made.

[jdconrad]

Another failure (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/722/console)