:x-pack:plugin:yamlRestTest - hundreds of failures (with specific seed)

[jakelandis]

:x-pack:plugin:yamlRestTest with the right seed can result in hundreds of failures. It seems that this happens when the ml yamlRestTests are executed prior to the data_stream yamlRestTests. This is likely a test not cleaning up after itself. (This is in the x-pack rest tests where a single cluster instance runs hundreds of tests in the same instance)

Build scan:

https://gradle-enterprise.elastic.co/s/gibqwc6eawv4c

Repro line:

git checkout dc0c9942de4594b852acfbc0ead73cc1fc0ce34b 
./gradlew :x-pack:plugin:yamlRestTest -Dtests.seed=275789FA3CA9B3DA --info

or more specifically:

git checkout dc0c9942de4594b852acfbc0ead73cc1fc0ce34b 
./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_stream/*}" --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/*}"  -Dtests.seed=275789FA3CA9B3DA --info

Reproduces locally?:
yes!

Applicable branches:
Discovered on master.

Failure history:
N/A - the task name was recently changed.

Failure excerpt:
The first failure:

1> [2020-09-09T23:24:26,341][INFO ][o.e.x.t.r.XPackRestIT    ] [test] [p0=data_stream/80_resolve_index_data_streams/Resolve index with hidden and closed indices] after test
2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_stream/80_resolve_index_data_streams/Resolve index with hidden and closed indices}" -Dtests.seed=275789FA3CA9B3DA -Dtests.security.manager=true -Dtests.locale=sr-Latn -Dtests.timezone=Asia/Calcutta -Druntime.java=11 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*
2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=data_stream/80_resolve_index_data_streams/Resolve index with hidden and closed indices}" -Dtests.seed=275789FA3CA9B3DA -Dtests.security.manager=true -Dtests.locale=sr-Latn -Dtests.timezone=Asia/Calcutta -Druntime.java=11 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*
2> java.lang.AssertionError: Failure at [data_stream/80_resolve_index_data_streams:132]: indices.3.name didn't match expected value:
                indices.3.name: expected String [test_index1] but was String [.ml-annotations-6]

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Data streams)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[martijnvg]

I was unable to reproduce this locally. Did something changes recently that doesn't make this reproduce easily?

I've opened #62205 to stop the cascading failure effect for yaml tests. So that next time only the problematic data stream yaml fails.

I think the Resolve index with hidden and closed indices yaml test should either remove .ml indices prior to running the resolve index api or the name param to resolve index api should be changed from * to *,-.ml-* (so that ml system indices are excluded). I think we should do the latter.

[droberts195]

This seems like the same problem as #38952. That was fixed in #39049 by making the cleanup in between tests more advanced, but only for one test suite, which was not the X-Pack REST tests.

I suspect the reason that failures are being seen now in the X-Pack REST tests is that up to now the .ml-annotations-6 index could have been created in some non-ML tests, but none of them cared. But now, presumably a data streams test has recently been added that asserts that no ML indices exist, so it fails if .ml-annotations-6 is created while it's running (as an async effect of the previous test).

I think we should do the latter.

I agree. Doing the former would not necessarily prevent the problem, as the index could be created in between the delete and the resolve.

The other alternative would be to copy the special cleanup logic from MlSingleNodeTestCase into XPackRestTest. However, there are two downsides to this:

1. The language clients that test against these REST tests have to duplicate special cleanup logic
2. It slows down running of the test suite

So if only one test is affected by the presence of an unexpected ML index then it will cause much less work across the company to work around it in that one test rather than changing the cleanup code.