Uh oh!
There was an error while loading. Please reload this page.
HBASE-25835 Ignore duplicate split requests from regionserver reports - #3218
Merged
Conversation
apurtell
commented
May 2, 2021
ContributorAuthor
Tested in an integration cluster test scenario (see #3208) but let's see what the unit test results in the CR report looks like. |
Apache-HBase
commented
May 2, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 3, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 3, 2021
🎊 +1 overall
This message was automatically generated. |
bharathv
approved these changes
May 3, 2021
| ", parent=" + parent + " splitKey=" + Bytes.toStringBinary(splitKey)); | ||
| } | ||
| master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent, splitKey)); | ||
| if (regionStates.getRegionState(parent).isOpened() && |
Contributor
There was a problem hiding this comment.
This is still on a best effort basis right? Theoretically duplicate requests are still possible because the region state in AM (from previous request) can be updated after a duplicate procedure is submitted thus passing this check, although that is less likely..
Contributor
There was a problem hiding this comment.
Agree. Let's add a comment here to say that this is not perfect, and is only used to reduce the concerns for operators. So later developers will know that the actual fencing is in other places.
And please get the RegionState once and store it a local variable, and then test isOpened and isSplitting on the same variable? And do we need to check whether it is null?
Contributor
There was a problem hiding this comment.
And do we need to check whether it is null?
Just to understand better, while creating Split Procedure, regionStates.getRegionState(parent) returning null should not happen right because we are trying to split an existing region (must be present in AM)? Or did I miss some race condition?
Contributor
There was a problem hiding this comment.
You should also not submit the request twice right? Race condition could happen everywhere. If the region has already been split and this is a delayed request then it could be null?
Contributor
There was a problem hiding this comment.
If the request is delayed to the point where parent is not only just successfully split but also removed after no refCount existing on it, then yes you are right, this is possible. Good to cover null check also.
Contributor
There was a problem hiding this comment.
Do we need !regionStates.getRegionState(parent).isSplitting() check here? When region state is OPEN it will be always true.
ContributorAuthor
There was a problem hiding this comment.
This addresses a case I've seen in real life. It's meant to fix this corner case to make the logs and operation less messy.
ContributorAuthor
There was a problem hiding this comment.
Ah, now github refreshes and I see all the other comments. Ok, will improve this. Just a moment...
Apache9
requested changes
May 3, 2021
| ", parent=" + parent + " splitKey=" + Bytes.toStringBinary(splitKey)); | ||
| } | ||
| master.getMasterProcedureExecutor().submitProcedure(createSplitProcedure(parent, splitKey)); | ||
| if (regionStates.getRegionState(parent).isOpened() && |
Contributor
There was a problem hiding this comment.
Agree. Let's add a comment here to say that this is not perfect, and is only used to reduce the concerns for operators. So later developers will know that the actual fencing is in other places.
And please get the RegionState once and store it a local variable, and then test isOpened and isSplitting on the same variable? And do we need to check whether it is null?
Processing of the RS report happens asynchronously from other activities which can mutate region state. For example, a split procedure may already be running. A split procedure cannot succeed if the parent region is no longer open, so we can ignore it in that case. Note that submitting more than one split procedure for a given region is harmless -- the split is fenced in the procedure handling -- but it would be noisy in the logs. Only one procedure can succeed. The other procedure(s) would abort during initialization and report failure with WARN level logging.
apurtell
commented
May 3, 2021
ContributorAuthor
Updated after feedback. |
Apache-HBase
commented
May 3, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 3, 2021
🎊 +1 overall
This message was automatically generated. |
pankaj72981
approved these changes
May 3, 2021
Apache-HBase
commented
May 3, 2021
🎊 +1 overall
This message was automatically generated. |
apurtell
commented
May 4, 2021
ContributorAuthor
@Apache9 please let me know if the latest patch addresses your change requests. |
apurtell
commented
May 4, 2021
ContributorAuthor
There are approvals and I believe all feedback has been addressed. Merging. We can reopen/revert/improve if necessary. |
asfgit pushed a commit
that referenced
this pull request
May 4, 2021
…#3218) Processing of the RS report happens asynchronously from other activities which can mutate region state. For example, a split procedure may already be running. A split procedure cannot succeed if the parent region is no longer open, so we can ignore it in that case. Note that submitting more than one split procedure for a given region is harmless -- the split is fenced in the procedure handling -- but it would be noisy in the logs. Only one procedure can succeed. The other procedure(s) would abort during initialization and report failure with WARN level logging. Signed-off-by: Bharath Vissapragada <bharathv@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Pankaj <pankajkumar@apache.org>
asfgit pushed a commit
that referenced
this pull request
May 4, 2021
…#3218) Processing of the RS report happens asynchronously from other activities which can mutate region state. For example, a split procedure may already be running. A split procedure cannot succeed if the parent region is no longer open, so we can ignore it in that case. Note that submitting more than one split procedure for a given region is harmless -- the split is fenced in the procedure handling -- but it would be noisy in the logs. Only one procedure can succeed. The other procedure(s) would abort during initialization and report failure with WARN level logging. Signed-off-by: Bharath Vissapragada <bharathv@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org> Signed-off-by: Pankaj <pankajkumar@apache.org>
Apache9
commented
May 5, 2021
Contributor
A bit late, it is still holiday in China, not always online outside... +1. Thanks @apurtell , the comment is great. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A SplitTableRegionProcedure may already be running when a regionserver report is received that includes a split request. The outcome is multiple SplitTableRegionProcedure procedures scheduled for the split request, only one of which can succeed. The others error out.
Do not create a split procedure in response to a region state change report if the region is not open or already splitting.