Fix Concurrent Snapshot Create+Delete + Delete Index #61770
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We had a bug here were we put a `null` value into the shard assignment mapping when reassigning work after a snapshot delete had gone through. This only affects partial snaphots but essentially dead-locks the snapshot process. Closes elastic#61762
original-brownbear
added
>bug
:Distributed Coordination/Snapshot/Restore
|
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
elasticmachine
added
the
Team:Distributed (Obsolete)
| @@ -1715,8 +1723,7 @@ public static ClusterState updateWithSnapshots(ClusterState state, | |||
| IndexMetadata indexMetadata = metadata.index(indexName); | |||
| if (indexMetadata == null) { | |||
| // The index was deleted before we managed to start the snapshot - mark it as missing. | |||
| builder.put(new ShardId(indexName, IndexMetadata.INDEX_UUID_NA_VALUE, 0), | |||
| new SnapshotsInProgress.ShardSnapshotStatus(null, ShardState.MISSING, "missing index", null)); | |||
| builder.put(new ShardId(indexName, IndexMetadata.INDEX_UUID_NA_VALUE, 0), ShardSnapshotStatus.MISSING); | |||
There was a problem hiding this comment.
Before concurrent snapshots this spot would cover all possible scenarios because we'd only be dealing with shard ids for indices that still exist in the repo ever beyond this point. If an index was deleted after assignment then it would just fail in the SnapshotShardsService and things would work out that way.
But with concurrent snapshots where we could have indices deleted from under a queued up shard snapshot we have to explicitly deal with this situation.
|
Thanks Yannick! |
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Sep 1, 2020
We had a bug here were we put a `null` value into the shard assignment mapping when reassigning work after a snapshot delete had gone through. This only affects partial snaphots but essentially dead-locks the snapshot process. Closes elastic#61762
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this pull request
Sep 1, 2020
We had a bug here were we put a `null` value into the shard assignment mapping when reassigning work after a snapshot delete had gone through. This only affects partial snaphots but essentially dead-locks the snapshot process. Closes elastic#61762
original-brownbear
added a commit
that referenced
this pull request
Sep 1, 2020
We had a bug here were we put a `null` value into the shard assignment mapping when reassigning work after a snapshot delete had gone through. This only affects partial snaphots but essentially dead-locks the snapshot process. Closes #61762
original-brownbear
added a commit
that referenced
this pull request
Sep 1, 2020
We had a bug here were we put a `null` value into the shard assignment mapping when reassigning work after a snapshot delete had gone through. This only affects partial snaphots but essentially dead-locks the snapshot process. Closes #61762
|
Linking this to #56911 so that I can find it again in future. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We had a bug here that is new to
7.9were we put anullvalue into the shardassignment mapping when reassigning work after a snapshot delete
had gone through. This only affects partial snaphots but essentially
dead-locks the snapshot process.
Closes #61762