[CI] Test cat data frame analytics single job with header failing

[ywelsch]

Failed 3 times on 7.x in the past days

Build scan: https://gradle-enterprise.elastic.co/s/7vn6gnjkkthfy

Output:

org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=ml/data_frame_analytics_cat_apis/Test cat data frame analytics single job with header} FAILED
    org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:35015], URI [/_ml/data_frame/analytics/dfa-classification-job], status line [HTTP/1.1 503 Service Unavailable]
    {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
        at __randomizedtesting.SeedInfo.seed([22DBE7B704A1086E:AA8FD86DAA5D6596]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:293)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:263)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:269)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:237)
        at org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.deleteAllDataFrameAnalytics(MlRestTestStateCleaner.java:110)
        at org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.clearMlMetadata(MlRestTestStateCleaner.java:32)
        at org.elasticsearch.xpack.test.rest.XPackRestIT.clearMlState(XPackRestIT.java:222)
        at org.elasticsearch.xpack.test.rest.XPackRestIT.cleanup(XPackRestIT.java:206)

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[cbuescher]

This looks similar to the issue mentioned here. @droberts195 should this be fixed on 7.x or is this a different issue?
https://gradle-enterprise.elastic.co/s/xnrdv7s6r64ks

[droberts195]

Another failure in https://gradle-enterprise.elastic.co/s/rbuxasr7s4yug

It's most likely the TrainedModelStatsService asynchronously creating indices part way through other tests. In #60462 (comment) the problem occurred during cleanup. In this test failure it's occurring part way through the test. The _cat API doesn't mind the stats index don't exist at all, as they won't if none of the data frame analytics models have ever been started. But it gets tripped up if the stats index exists but isn't searchable.

[przemekwitek]

Happened again in https://gradle-enterprise.elastic.co/s/wmrnvevbdno7i