[CI] SharedClusterSnapshotRestoreIT.testAbortedSnapshotDuringInitDoesNotStart Fails

[original-brownbear]

Happened here:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/606/

  2> REPRODUCE WITH: ./gradlew :server:integTest -Dtests.seed=D7E1CF66C40F4360 -Dtests.class=org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT -Dtests.method="testAbortedSnapshotDuringInitDoesNotStart" -Dtests.security.manager=true -Dtests.locale=es-CL -Dtests.timezone=America/Punta_Arenas -Dcompiler.java=11 -Druntime.java=8

fails with:

ERROR   0.87s J1 | SharedClusterSnapshotRestoreIT.testAbortedSnapshotDuringInitDoesNotStart <<< FAILURES!
   > Throwable #1: UncategorizedExecutionException[Failed execution]; nested: ExecutionException[java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+master+internalClusterTest/server/build/testrun/integTest/J1/temp/org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT_D7E1CF66C40F4360-001/tempDir-002/repos/dJWCuVKIEy/indices/KbMMazL8QxmxzkgdNZRFeQ/meta-hJC9954EReKInUiOrlhLfg.dat]; nested: NoSuchFileException[/var/lib/jenkins/workspace/elastic+elasticsearch+master+internalClusterTest/server/build/testrun/integTest/J1/temp/org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT_D7E1CF66C40F4360-001/tempDir-002/repos/dJWCuVKIEy/indices/KbMMazL8QxmxzkgdNZRFeQ/meta-hJC9954EReKInUiOrlhLfg.dat];
   > 	at __randomizedtesting.SeedInfo.seed([D7E1CF66C40F4360:AF3C0690F8AC786E]:0)
   > 	at org.elasticsearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:101)
   > 	at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:62)
   > 	at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:34)
   > 	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:52)
   > 	at org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT.lambda$testAbortedSnapshotDuringInitDoesNotStart$24(SharedClusterSnapshotRestoreIT.java:3687)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:832)
   > 	at org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT.testAbortedSnapshotDuringInitDoesNotStart(SharedClusterSnapshotRestoreIT.java:3685)
   > 	at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.util.concurrent.ExecutionException: java.nio.file.NoSuchFileException: /var/lib/jenkins/workspace/elastic+elasticsearch+master+internalClusterTest/server/build/testrun/integTest/J1/temp/org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT_D7E1CF66C40F4360-001/tempDir-002/repos/dJWCuVKIEy/indices/KbMMazL8QxmxzkgdNZRFeQ/meta-hJC9954EReKInUiOrlhLfg.dat
   > 	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:266)
   > 	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:253)
   > 	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:87)
   > 	at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:57)
   > 	... 43 more

relates #38368 , #38226

---

Seems like it's the same failure that happened was supposed to be fixed in #38368 but at a much lower rate.

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

Seems like a problem with the mock repository block.

If I put a sleep before the busy assert:

            TimeUnit.SECONDS.sleep(2L);
            // The deletion must set the snapshot in the ABORTED state
            assertBusy(() -> {
                SnapshotsStatusResponse status =
                    client.admin().cluster().prepareSnapshotStatus("repository").setSnapshots("snap").get();
                assertThat(status.getSnapshots().iterator().next().getState(), equalTo(State.ABORTED));
            });

it fails every time, seems the snapshot is deleted already even without unblocking the master node in the next lines:

// Now unblock the repository
            unblockNode("repository", internalCluster().getMasterName());
            blocked = false;

            assertAcked(delete.get());
            expectThrows(SnapshotMissingException.class, () ->
                client.admin().cluster().prepareGetSnapshots("repository").setSnapshots("snap").get());

[cbuescher]

Failed again to day on 7.0: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+internalClusterTest/139/console