Several test failures in :x-pack:plugin:security:test in 6.3

[droberts195]

In https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=sles/86/console the following tests failed:

org.elasticsearch.xpack.security.authc.esnative.NativeRealmIntegTests testAddUserAndRoleThenAuth
org.elasticsearch.xpack.security.authz.IndexAliasesTests testGetAliasesCreateOnlyPermissionStrict
org.elasticsearch.xpack.security.authz.IndexAliasesTests testGetAliasesCreateOnlyPermissionStrict
org.elasticsearch.xpack.security.audit.logfile.AuditTrailSettingsUpdateTests testInvalidFilterSettings
org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests testThatTransportClientWithOnlyTruststoreCanConnectToNoClientAuthProfile
org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile
org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile

None of these reproduce individually. These are the REPRO commands:

./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.audit.logfile.AuditTrailSettingsUpdateTests -Dtests.method="testInvalidFilterSettings" -Dtests.security.manager=true -Dtests.locale=sr-Latn-RS -Dtests.timezone=America/Argentina/Rio_Gallegos
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests -Dtests.method="testThatTransportClientWithOnlyTruststoreCanConnectToNoClientAuthProfile" -Dtests.security.manager=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Chile/Continental
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests -Dtests.method="testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile" -Dtests.security.manager=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Chile/Continental
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests -Dtests.method="testThatTransportClientWithOnlyTruststoreCannotConnectToDefaultProfile" -Dtests.security.manager=true -Dtests.locale=sr-Latn-BA -Dtests.timezone=Chile/Continental
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.authc.esnative.NativeRealmIntegTests -Dtests.method="testAddUserAndRoleThenAuth" -Dtests.security.manager=true -Dtests.locale=el-CY -Dtests.timezone=Africa/Algiers
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.authz.IndexAliasesTests -Dtests.method="testGetAliasesCreateOnlyPermissionStrict" -Dtests.security.manager=true -Dtests.locale=ru-RU -Dtests.timezone=Africa/Malabo
./gradlew :x-pack:plugin:security:test -Dtests.seed=22E2031847F1522 -Dtests.class=org.elasticsearch.xpack.security.authz.IndexAliasesTests -Dtests.method="testGetAliasesCreateOnlyPermissionStrict" -Dtests.security.manager=true -Dtests.locale=ru-RU -Dtests.timezone=Africa/Malabo

There are some exceptions in the log file indicating SSL setup problems, for example:

  1> [2018-05-25T08:17:49,624][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node_sc2] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:56147, remoteAddress=/127.0.0.1:51564}], closing connection
  1> io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000064000000000000000108004d3603010d417574686f72697a6174696f6e2e4261736963206447567a64463931633256794f6e67746347466a617931305a584e304c58426863334e3362334a6b0016696e7465726e616c3a7463702f68616e647368616b6500
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000064000000000000000108004d3603010d417574686f72697a6174696f6e2e4261736963206447567a64463931633256794f6e67746347466a617931305a584e304c58426863334e3362334a6b0016696e7465726e616c3a7463702f68616e647368616b6500
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more

Also:

  1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  1> 	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  1> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_172]
  1> 	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  1> 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  1> 	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:?]
  1> 	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
  1> 	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
  1> 	at org.elasticsearch.xpack.core.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:594) ~[x-pack-core-6.3.0-SNAPSHOT.jar:6.3.0-SNAPSHOT]
  1> 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
  1> 	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
  1> 	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
  1> 	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_172]
  1> 	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:?]
  1> 	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
  1> 	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:?]
  1> 	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:?]
  1> 	at org.elasticsearch.xpack.core.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:594) ~[x-pack-core-6.3.0-SNAPSHOT.jar:6.3.0-SNAPSHOT]
  1> 	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ~[?:?]
  1> 	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more

And:

  1> [2018-05-25T08:17:38,683][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node_s1] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:30280, remoteAddress=/127.0.0.1:37174}], closing connection
  1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
  1> 	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  1> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_172]
  1> 	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
  1> 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
  1> 	at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1935) ~[?:?]
  1> 	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:237) ~[?:?]
  1> 	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:?]
  1> 	at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:?]
  1> 	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_172]
  1> 	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:?]
  1> 	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more
  1> [2018-05-25T08:17:38,694][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [programmatic_transport_client] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:37174, remoteAddress=/127.0.0.1:30280}], closing connection
  1> io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
  1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
  1> Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
  1> 	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
  1> 	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  1> 	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_172]
  1> 	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
  1> 	... 15 more

[elasticmachine]

Pinging @elastic/es-security

[jkakavas]

This one looks like the node attempted to communicate over http

455300000064000000000000000108004d3603010d417574686f72697a6174696f6e2e4261736963206447567a64463931633256794f6e67746347466a617931305a584e304c58426863334e3362334a6b0016696e7465726e616c3a7463702f68616e647368616b6500

-->

ESd��M6��
Authorization.Basic dGVzdF91c2VyOngtcGFjay10ZXN0LXBhc3N3b3Jk�internal:tcp/handshake

and dGVzdF91c2VyOngtcGFjay10ZXN0LXBhc3N3b3Jk is Base64 encoded test_user:x-pack-test-password

None of these tests have failed since, but keeping this open until I ca n figure out what could have caused the failure

[imotov]

I cannot quite figure out if this is the same issue or something completely different:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=sles/174/consoleText

[ywelsch]

More of this?
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-windows-compatibility/177

[danielmitterdorfer]

Another instance in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/209/console

The reproduction line is:

./gradlew :x-pack:plugin:security:test \
  -Dtests.seed=B0DEA59D380A135F \
  -Dtests.class=org.elasticsearch.xpack.security.transport.ssl.SslMultiPortTests \
  -Dtests.method="testThatTransportClientWithOnlyTruststoreCannotConnectToClientProfile" \
  -Dtests.security.manager=true \
  -Dtests.locale=sr-RS \
  -Dtests.timezone=US/Pacific

[cbuescher]

@jkakavas I got a recent one here:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=debian/87/consoleFull

Doesn't reproduce, but I guess we don't mute these to gather more info and they fail not too often?

[aaparent]

Hi,
Any news regarding this issue?
I'm having this one:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000027000000000000000208004d3603000016696e7465726e616c3a7463702f68616e647368616b6500
but I don't have a clue why.

I'm using ES6.4.2 with Xpack and the Java TransportClient

[jkakavas]

Hi @aparent-talend .

This issue is for a test that fails in our CI.

The simple explanation for your error is that you have Elasticsearch configured to use TLS on the transport layer but you haven't configured your Transport Client accordingly so it tries to communicate over plain TCP. If that is the case, please open an issue in our forums and we will do our best to help you there.

[aaparent]

Thanks a lot, sorry for bothering you on the wrong place :-)