CI failures due to indices not being cleaned up as expected #46091
Description
There appears to be some kind of problem with the cleanup code that ensures all indices are cleaned up between tests. It's difficult to be sure, but there seems to be a trend of a wide variety of tests failing because of a ResourceAlreadyExistsException when creating an index with no visible cause in the test itself.
I noticed this in this master intake build, which is IndexLifecycleIT.testStartStopILM, which creates an index as one of the first operations it does - the cluster should be blank at this point.
Looking at build stats for failures with messages containing resource_already_exists_exception, we can see it was very quiet prior to July 23, and has built more and more, suggesting this may be related to recent build changes:
[Note: This graph excludes a 10 minute period on Aug. 15 which had 608 failures, to make the scale clearer]
There is no clear relation between the tests which fail due to indices already existing that shouldn't and a specific functional area. It appears to happen most often to client and YAML tests, although not exclusively. In a few spot checks I've seen:
IndexLifecycleIT.testStartStopILMCCSDuelIT.testPaginationtest {yaml=search/240_date_nanos/date_nanos requires dates after 1970 and before 2262}IndicesClientIT.testCloseExistingIndexDatafeedJobsRestIT.testLookbackWithGeo
All of these appear to have failed while creating indices when the cluster should be a blank slate.