[ML] Async creation of the .ml-annotations index can spill into subsequent tests

[droberts195]

Following the change of #38903 the way we create the .ml-annotations-6 index will cause occasional assertion failures in tests that extend ESSingleNodeTestCase. One such failure is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+internalClusterTest/589/consoleText

Currently the .ml-annotations-6 index is created by a cluster state listener if any other ML index exists. This is nice for production, because it avoids the clutter of unconditionally creating .ml-annotations-6 if ML is not in use, yet makes sure it is available for creating annotations as soon as any ML functionality is used. But it is nasty for tests, as creation of the index can spill from one test into another.

Some ideas for fixing this could be:

• Switch all ML tests that extend ESSingleNodeTestCase to instead extend an intermediate ML test base class that has extra cleanup code. The extra cleanup would need to:
  1. Check if any ML indices existed
  2. If so, wait for the .ml-annotations-6 index to exist
  3. Delete the other ML indices found in step 1.
  4. Call the super class cleanup

or

• Add a setting to disable creation of the ML annotations index and aliases. Set this in ESSingleNodeTestCase. Any ML tests that were testing ML annotations would have to explicitly create it.

There might be better ways. Any thoughts?

[elasticmachine]

Pinging @elastic/ml-core

[jasontedor]

I favor the first approach. The second approach would require that the setting be in OSS to put it into ESSingleNodeTestCase, or that we have a single base class that all X-Pack ESSingleNodeTestCases would extend to set the setting there. Neither of those options are attractive to me. The first is not attractive because of the spilling into OSS, and the second is not attractive because of the overhead involved.

[iverase]

Another case:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=sles-12/245/console