Failure in ML BwC REST Tests

[jdconrad]

NodeDisconnectedException failure, unable to reproduce locally, but seeing this error a number of times today. Wondering if there is some relation to (#31834).

Reproduce:

  -Dtests.seed=896067E457C12CA0 \
  -Dtests.security.manager=true \
  -Dtests.locale=bas \
  -Dtests.timezone=Africa/Luanda \
  -Dcompiler.java=12 \
  -Druntime.java=11 \
  -Dtests.rest.suite=mixed_cluster \
  -Dtests.rest.blacklist=mixed_cluster/10_basic/Start scroll in mixed cluster on upgraded node that we will continue after upgrade,mixed_cluster/30_ml_jobs_crud/Create a job in the mixed cluster and write some data,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed in mixed cluster,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed without aggs in mixed cluster,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed with aggs in mixed cluster,mixed_cluster/80_data_frame_jobs_crud/Test put batch data frame transforms on mixed cluster,mixed_cluster/80_data_frame_jobs_crud/Test put continuous data frame transform on mixed cluster

Build:
https://scans.gradle.com/s/32bmadif5ad4e/console-log

Error:

org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT > test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster categorization job} FAILED
--
java.lang.AssertionError: Failure at [mixed_cluster/30_ml_jobs_crud:80]: expected [2xx] status code but api [ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"}],"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"},"status":500}]
 
Caused by:
java.lang.AssertionError: expected [2xx] status code but api [ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"}],"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"},"status":500}]
REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:v7.4.0#oneThirdUpgradedTestRunner --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster categorization job}" -Dtests.seed=896067E457C12CA0 -Dtests.security.manager=true -Dtests.locale=bas -Dtests.timezone=Africa/Luanda -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.suite=mixed_cluster -Dtests.rest.blacklist=mixed_cluster/10_basic/Start scroll in mixed cluster on upgraded node that we will continue after upgrade,mixed_cluster/30_ml_jobs_crud/Create a job in the mixed cluster and write some data,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed in mixed cluster,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed without aggs in mixed cluster,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed with aggs in mixed cluster,mixed_cluster/80_data_frame_jobs_crud/Test put batch data frame transforms on mixed cluster,mixed_cluster/80_data_frame_jobs_crud/Test put continuous data frame transform on mixed cluster
 
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT > test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job} FAILED
java.lang.AssertionError: Failure at [mixed_cluster/30_ml_jobs_crud:29]: expected [2xx] status code but api [ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"}],"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"},"status":500}]
 
Caused by:
java.lang.AssertionError: expected [2xx] status code but api [ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"}],"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:43380][cluster:admin/xpack/ml/job/close] disconnected]\n"},"status":500}]
REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:v7.4.0#oneThirdUpgradedTestRunner --tests "org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT.test {p0=mixed_cluster/30_ml_jobs_crud/Test get old cluster job}" -Dtests.seed=896067E457C12CA0 -Dtests.security.manager=true -Dtests.locale=bas -Dtests.timezone=Africa/Luanda -Dcompiler.java=12 -Druntime.java=11 -Dtests.rest.suite=mixed_cluster -Dtests.rest.blacklist=mixed_cluster/10_basic/Start scroll in mixed cluster on upgraded node that we will continue after upgrade,mixed_cluster/30_ml_jobs_crud/Create a job in the mixed cluster and write some data,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed in mixed cluster,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed without aggs in mixed cluster,mixed_cluster/40_ml_datafeed_crud/Put job and datafeed with aggs in mixed cluster,mixed_cluster/80_data_frame_jobs_crud/Test put batch data frame transforms on mixed cluster,mixed_cluster/80_data_frame_jobs_crud/Test put continuous data frame transform on mixed cluster
 ```

[elasticmachine]

Pinging @elastic/ml-core

[ywelsch]

Another test failure here: https://gradle.com/s/ookekq4pn267k

This also reproduces locally.

Relevant output:

1> [2019-07-29T06:50:55,147][INFO ][o.e.u.UpgradeClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "error" : {
  1>         "root_cause" : [
  1>           {
  1>             "type" : "node_disconnected_exception",
  1>             "reason" : "[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected",
  1>             "stack_trace" : "NodeDisconnectedException[[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected]
  1> "
  1>           }
  1>         ],
  1>         "type" : "node_disconnected_exception",
  1>         "reason" : "[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected",
  1>         "stack_trace" : "NodeDisconnectedException[[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected]
  1> "
  1>       },
  1>       "status" : 500
  1>     }
  1>   }
  1> }]


2> java.lang.AssertionError: Failure at [mixed_cluster/30_ml_jobs_crud:29]: expected [2xx] status code but api [ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected]\n"}],"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected]\n"},"status":500}]

        Caused by:
        java.lang.AssertionError: expected [2xx] status code but api [ml.close_job] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected]\n"}],"type":"node_disconnected_exception","reason":"[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected","stack_trace":"NodeDisconnectedException[[node-1][127.0.0.1:38311][cluster:admin/xpack/ml/job/close] disconnected]\n"},"status":500}]

[2019-07-29T06:50:53,597][WARN ][o.e.t.TcpTransport       ] [upgraded-node-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:48210, remoteAddress=127.0.0.1/127.0.0.1:38311}], closing connection
java.lang.IllegalStateException: Message not fully read (response) for requestId [490], handler [org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler/org.elasticsearch.action.support.tasks.TransportTasksAction$AsyncAction$1@698b95f9], error [false]; resetting
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:147) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:104) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:661) [elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62) [transport-netty4-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) [netty-handler-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

[przemekwitek]

I've just merged #44905 into master.
This PR should fix the issue.