[CI] SharedClusterSnapshotRestoreIT testDeletionOfFailingToRecoverIndexShouldStopRestore failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #21263 / part1
• elasticsearch-pull-request #66140 / part-1
• elasticsearch-intake #21259 / part1
• elasticsearch-pull-request #66140 / part-1
• elasticsearch-pull-request #66126 / part-1
• elasticsearch-pull-request #66078 / part-1

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT.testDeletionOfFailingToRecoverIndexShouldStopRestore" -Dtests.seed=127C6CFF1D1C565D -Dtests.locale=fr-BE -Dtests.timezone=Asia/Yakutsk -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.snapshots.SnapshotRestoreException: [default:test-repo:test-snap/ys7WgC4oQfaufdfgx_fumg] cannot restore index [test-idx] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name

Issue Reasons:

• [main] 6 failures in test testDeletionOfFailingToRecoverIndexShouldStopRestore (3.3% fail rate in 180 executions)
• [main] 2 failures in step part1 (9.5% fail rate in 21 executions)
• [main] 4 failures in step part-1 (5.0% fail rate in 80 executions)
• [main] 2 failures in pipeline elasticsearch-intake (9.5% fail rate in 21 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (3.8% fail rate in 80 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 failures in test testDeletionOfFailingToRecoverIndexShouldStopRestore (2.2% fail rate in 178 executions)
• [main] 3 failures in step part-1 (3.7% fail rate in 81 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (3.7% fail rate in 82 executions)

Build Scans:

• elasticsearch-intake #21259 / part1
• elasticsearch-pull-request #66140 / part-1
• elasticsearch-pull-request #66126 / part-1
• elasticsearch-pull-request #66078 / part-1

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[pxsalehi]

This doesn't seem like a blocker to me.

[ywangd]

This failure is caused by #125652 which changed GetIndexAction to run on the local node. This means it can see an index that is not yet visible to the master. The sequence of the test steps are:

1. Restore the index async
2. Get the Index as a check to for start of the restore
3. Delete the index
4. Wait for restore to fail
5. Restore one more time for success

The issue is that 2 returns before master knows about the index. And delete index (cluster.wipeIndices) silently ignores 404. Therefore the index is not deleted and step 5 fails because it cannot restore on a non-closed index.

It is easy to just fix this test, e.g. by making sure step 2 talks to the master node. But I wonder similar failures could happen elsewhere. And more fundamentally, maybe we should not change GetIndex to run on local node by default since it feels like a slight breaking change (doc)? What you do think? @nielsbauman @DaveCTurner

[DaveCTurner]

I think it's right to run GetIndexAction on the local node, but we can fix the doc to match. Even when running on the master there is a chance of stale reads (we do not validate that the responding master is in fact the master before responding) so running it on the local node doesn't weaken any guarantees in this area.

[DaveCTurner]

e.g. by making sure step 2 talks to the master node.

... or by just inserting a GET _cluster/health?wait_for_events=languid

[ywangd]

running it on the local node doesn't weaken any guarantees in this area.

For my knowledge: For reads made on the master node, even if the master node fails after the initial read, I think we can be sure that the same value is carried on by the new master? That is, if we read again, we can be sure the value stays the same assuming a single client access which is the case here. But this test failure means we read some value, but reading again does not find it. IIUC, this is not possible with a master node read action? So it seems to me that it does weaken the guarantee?