HBASE-25835 Ignore duplicate split requests from regionserver reports #3218
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tested in an integration cluster test scenario (see #3208) but let's see what the unit test results in the CR report looks like. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
bharathv
approved these changes
May 3, 2021
| @@ -1126,7 +1126,13 @@ private void updateRegionSplitTransition(final ServerName serverName, final Tran | |||
| LOG.debug("Split request from " + serverName + | |||
| ", parent=" + parent + " splitKey=" + Bytes.toStringBinary(splitKey)); | |||
| } | |||
| master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent, splitKey)); | |||
| if (regionStates.getRegionState(parent).isOpened() && | |||
There was a problem hiding this comment.
This is still on a best effort basis right? Theoretically duplicate requests are still possible because the region state in AM (from previous request) can be updated after a duplicate procedure is submitted thus passing this check, although that is less likely..
There was a problem hiding this comment.
Agree. Let's add a comment here to say that this is not perfect, and is only used to reduce the concerns for operators. So later developers will know that the actual fencing is in other places.
And please get the RegionState once and store it a local variable, and then test isOpened and isSplitting on the same variable? And do we need to check whether it is null?
There was a problem hiding this comment.
And do we need to check whether it is null?
Just to understand better, while creating Split Procedure, regionStates.getRegionState(parent) returning null should not happen right because we are trying to split an existing region (must be present in AM)? Or did I miss some race condition?
There was a problem hiding this comment.
You should also not submit the request twice right? Race condition could happen everywhere. If the region has already been split and this is a delayed request then it could be null?
There was a problem hiding this comment.
If the request is delayed to the point where parent is not only just successfully split but also removed after no refCount existing on it, then yes you are right, this is possible. Good to cover null check also.
There was a problem hiding this comment.
Do we need !regionStates.getRegionState(parent).isSplitting() check here? When region state is OPEN it will be always true.
There was a problem hiding this comment.
This addresses a case I've seen in real life. It's meant to fix this corner case to make the logs and operation less messy.
There was a problem hiding this comment.
Ah, now github refreshes and I see all the other comments. Ok, will improve this. Just a moment...
Apache9
requested changes
May 3, 2021
| @@ -1126,7 +1126,13 @@ private void updateRegionSplitTransition(final ServerName serverName, final Tran | |||
| LOG.debug("Split request from " + serverName + | |||
| ", parent=" + parent + " splitKey=" + Bytes.toStringBinary(splitKey)); | |||
| } | |||
| master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent, splitKey)); | |||
| if (regionStates.getRegionState(parent).isOpened() && | |||
There was a problem hiding this comment.
Agree. Let's add a comment here to say that this is not perfect, and is only used to reduce the concerns for operators. So later developers will know that the actual fencing is in other places.
And please get the RegionState once and store it a local variable, and then test isOpened and isSplitting on the same variable? And do we need to check whether it is null?
Processing of the RS report happens asynchronously from other activities which can mutate region state. For example, a split procedure may already be running. A split procedure cannot succeed if the parent region is no longer open, so we can ignore it in that case. Note that submitting more than one split procedure for a given region is harmless -- the split is fenced in the procedure handling -- but it would be noisy in the logs. Only one procedure can succeed. The other procedure(s) would abort during initialization and report failure with WARN level logging.
|
Updated after feedback. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
pankaj72981
approved these changes
May 3, 2021
|
🎊 +1 overall
This message was automatically generated. |
|
@Apache9 please let me know if the latest patch addresses your change requests. |
|
There are approvals and I believe all feedback has been addressed. Merging. We can reopen/revert/improve if necessary. |
asfgit
pushed a commit
that referenced
this pull request
May 4, 2021
…#3218) Processing of the RS report happens asynchronously from other activities which can mutate region state. For example, a split procedure may already be running. A split procedure cannot succeed if the parent region is no longer open, so we can ignore it in that case. Note that submitting more than one split procedure for a given region is harmless -- the split is fenced in the procedure handling -- but it would be noisy in the logs. Only one procedure can succeed. The other procedure(s) would abort during initialization and report failure with WARN level logging. Signed-off-by: Bharath Vissapragada <bharathv@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Pankaj <pankajkumar@apache.org>
asfgit
pushed a commit
that referenced
this pull request
May 4, 2021
…#3218) Processing of the RS report happens asynchronously from other activities which can mutate region state. For example, a split procedure may already be running. A split procedure cannot succeed if the parent region is no longer open, so we can ignore it in that case. Note that submitting more than one split procedure for a given region is harmless -- the split is fenced in the procedure handling -- but it would be noisy in the logs. Only one procedure can succeed. The other procedure(s) would abort during initialization and report failure with WARN level logging. Signed-off-by: Bharath Vissapragada <bharathv@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Pankaj <pankajkumar@apache.org>
|
A bit late, it is still holiday in China, not always online outside... +1. Thanks @apurtell , the comment is great. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A SplitTableRegionProcedure may already be running when a regionserver report is received that includes a split request. The outcome is multiple SplitTableRegionProcedure procedures scheduled for the split request, only one of which can succeed. The others error out.
Do not create a split procedure in response to a region state change report if the region is not open or already splitting.