[CI] Failure in ml.integration.RegressionIT.testStopAndRestart

[talevy]

Something has triggered this test to fail 14 times in CI, starting on October 2.

reproduction step:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.RegressionIT.testStopAndRestart" -Dtests.seed=6C78F050FC5C2A2

Failed with stacktrace:

java.lang.AssertionError: 

Expected: <stopped>
     but: was <failed>

Open stacktrace

[2019-10-04T16:15:56,253][ERROR][o.e.x.m.i.RegressionIT   ] [testStopAndRestart] Failed to stop data frame analytics jobs; trying force
org.elasticsearch.ElasticsearchStatusException: cannot close data frame analytics [regression_stop_and_restart] because it failed, use force stop instead
	at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.conflictStatusException(ExceptionsHelper.java:59) ~[x-pack-core-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.action.TransportStopDataFrameAnalyticsAction.findAnalyticsToStop(TransportStopDataFrameAnalyticsAction.java:127) ~[x-pack-ml-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.action.TransportStopDataFrameAnalyticsAction.lambda$doExecute$1(TransportStopDataFrameAnalyticsAction.java:102) ~[x-pack-ml-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.xpack.ml.action.TransportStopDataFrameAnalyticsAction.lambda$expandIds$4(TransportStopDataFrameAnalyticsAction.java:173) ~[x-pack-ml-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[dimitris-athanasiou]

@talevy Could you please add links to CI and the build scans?

[droberts195]

Here’s one example:

Jenkins: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+g1gc/362/console
Scan: https://gradle-enterprise.elastic.co/s/bdbipsekpfijw

[droberts195]

Looks like it’s due to:

18:45:24 » ERROR][o.e.x.m.p.AbstractNativeProcess] [integTest-1] [regression_stop_and_restart] analytics process stopped unexpectedly: Input error: expected no more than '232' rows but got '350' rows. Please report this problem.

[dimitris-athanasiou]

This is really unexpected. I'll look into how this could happen.

[droberts195]

It just failed again, but on a different assertion and no message about the C++ process exiting early (unless I missed it - possible as I looked on a phone).

Details are:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/430/console
• https://gradle-enterprise.elastic.co/s/me2mnmmofdfky

[dimitris-athanasiou]

I'll mute it for now.