SnapshotLifecycleRestIT testFullPolicySnapshot failure

[davidkyle]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-windows-compatibility/os=windows-2016/331/console
https://gradle-enterprise.elastic.co/s/fmz3orwsf6ceu

org.elasticsearch.client.ResponseException: 
method [DELETE], host [http://127.0.0.1:60674], URI [/_snapshot/my-repo/snap-dgpzyeujtlil-pxstwwurg], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"concurrent_snapshot_execution_exception","reason":"[my-repo:snap-dgpzyeujtlil-pxstwwurg/5U_JWp2pRceMG2VKwqteqQ] another snapshot is currently running cannot delete"}],"type":"concurrent_snapshot_execution_exception","reason":"[my-repo:snap-dgpzyeujtlil-pxstwwurg/5U_JWp2pRceMG2VKwqteqQ] another snapshot is currently running cannot delete"},"status":503}

at __randomizedtesting.SeedInfo.seed([A77F6987797E068:3AE5B97574CB83D]:0)
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
at org.elasticsearch.test.rest.ESRestTestCase.wipeSnapshots(ESRestTestCase.java:613)
at org.elasticsearch.test.rest.ESRestTestCase.lambda$wipeCluster$3(ESRestTestCase.java:504)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:854)
at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:503)
at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:287)
...

Does not reproduce:

gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testFullPolicySnapshot" \
  -Dtests.seed=A77F6987797E068 \
  -Dtests.security.manager=true \
  -Dtests.locale=ca \
  -Dtests.timezone=Europe/Ulyanovsk \
  -Dcompiler.java=13

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

[davidkyle]

A handful of similar failures:

https://gradle-enterprise.elastic.co/s/scv4m7g4w7kcs
https://gradle-enterprise.elastic.co/s/ttguod4m35fqs

Muted on master in ae90f8a

[danielmitterdorfer]

We have another failure in 7.x (CI link, build scan):

The symptoms look a bit different to me though:

java.lang.AssertionError: 
Expected: a string starting with "snap-"
     but: was "snapuyytvuakl-31toqfartrwfolzgmlvdra"
	at __randomizedtesting.SeedInfo.seed([D4BE8BA449B8A835:DD6726AB6963F060]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.lambda$testFullPolicySnapshot$1(SnapshotLifecycleRestIT.java:110)

reproduction line:

./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testFullPolicySnapshot" -Dtests.seed=D4BE8BA449B8A835 -Dtests.security.manager=true -Dtests.locale=ko-KR -Dtests.timezone=Canada/Central -Dcompiler.java=13 -Druntime.java=8

Does not reproduce locally.