[CI] Test Failure ReloadSecureSettingsIT testWrongKeystorePassword

[jdconrad]

Console Link:
(https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/200/console)

Reproduce With:

./gradlew :server:integTest \
  -Dtests.seed=E75D0D0620B36748 \
  -Dtests.class=org.elasticsearch.action.admin.ReloadSecureSettingsIT \
  -Dtests.method="testWrongKeystorePassword" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-MX \
  -Dtests.timezone=America/Fortaleza

Stack Trace:

16:18:37 FAILURE 0.29s J0 | ReloadSecureSettingsIT.testWrongKeystorePassword <<< FAILURES!
16:18:37    > Throwable #1: java.lang.AssertionError: 
16:18:37    > Expected: an instance of java.lang.SecurityException
16:18:37    >      but: <java.io.IOException: javax.crypto.AEADBadTagException: Error finalising cipher data: mac check in GCM failed> is a java.io.IOException
16:18:37    > 	at __randomizedtesting.SeedInfo.seed([E75D0D0620B36748:3A84FDB91FFEC9B7]:0)
16:18:37    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
16:18:37    > 	at org.elasticsearch.action.admin.ReloadSecureSettingsIT$4.onResponse(ReloadSecureSettingsIT.java:211)
16:18:37    > 	at org.elasticsearch.action.admin.ReloadSecureSettingsIT$4.onResponse(ReloadSecureSettingsIT.java:199)
16:18:37    > 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:66)
16:18:37    > 	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:62)
16:18:37    > 	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:245)
16:18:37    > 	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:222)
16:18:37    > 	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:146)
16:18:37    > 	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:199)
16:18:37    > 	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
16:18:37    > 	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1059)
16:18:37    > 	at org.elasticsearch.transport.TcpTransport$2.doRun(TcpTransport.java:1571)
16:18:37    > 	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
16:18:37    > 	at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)
16:18:37    > 	at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1563)
16:18:37    > 	at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1519)
16:18:37    > 	at org.elasticsearch.transport.TcpTransport.consumeNetworkReads(TcpTransport.java:1328)
16:18:37    > 	at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:191)
16:18:37    > 	at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:213)
16:18:37    > 	at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:54)
16:18:37    > 	at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:119)
16:18:37    > 	at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:355)
16:18:37    > 	at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:216)
16:18:37    > 	at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:144)
16:18:37    > 	at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:109)
16:18:37    > 	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-core-infra

[jaymode]

@jkakavas this may be related to FIPS. Can you take a look?

[jkakavas]

I had it in my radar since yesterday ( https://groups.google.com/a/elastic.co/d/msg/build-elasticsearch/KMDL9V3yJi0/NGiG3OH1BwAJ ) . This is because of the change introduced here #31989 (comment) , and it looks that the behavior with BC is not consistent. I will take a closer look and adjust.

[droberts195]

This failed again in
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java8fips,nodes=virtual&&linux/207/console

Since it's FIPS-only I haven't muted it as it won't interfere with PR builds or intake builds, but let me know if you'd prefer me to mute it.

[jkakavas]

Thanks @droberts195 , I agree this is not worth muting. I will address this today.

[jkakavas]

This problem is not entirely specific to this test, but this is the only test where we load a KeyStoreWrapper with a wrong password, hence it manifested here. I raised #32464 to fix the underlying issue, that will also mitigate the test failures here