Uh oh!
There was an error while loading. Please reload this page.
HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor - #3234
Merged
Conversation
apurtell
commented
May 5, 2021
ContributorAuthor
This change is moved here from #3230. We know from the test report there that |
apurtell
commented
May 5, 2021
ContributorAuthor
@Apache9 added some context on #3230
|
ContributorAuthor
Although the unit test will fail, when I tried this change out on a cluster in a very write heavy ingestion test, the end result was good. The ingestion test completes successfully and without weird artifacts in the logging. All split regions are GCed by procedures. In memory state aligns with filesystem state. |
Apache-HBase
commented
May 5, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 5, 2021
🎊 +1 overall
This message was automatically generated. |
saintstack
approved these changes
May 5, 2021
saintstack
left a comment
Contributor
There was a problem hiding this comment.
Yes to one-place only (in the procedure).
I've lost context on why the CJ redundant removes. I defer to the test you ran Andrew at scale.
(I just noticed the wonky split message -- have started a bit of ITBLL over here...)
Apache-HBase
commented
May 5, 2021
💔 -1 overall
This message was automatically generated. |
ContributorAuthor
I've committed a few changes over the past couple of days that address two issues here. One was an accumulation of SPLIT state regionStates for as long as CatalogJanitor is deferring cleanup, because daughters still have references, because compaction is backed up. The accumulation is fine but it caused concerning warnings at each balancer iteration. Another was a race condition between split handling and regionserver report handling that could cause multiple split procedures to get scheduled for the same split request. Only one could succeed. The others would cause log noise but their failures were harmless. Would be good to get a second opinion. |
apurtell
commented
May 6, 2021
ContributorAuthor
Ok, let me rebase this and fix the unit tests. |
…itor In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures.
Apache-HBase
commented
May 6, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 6, 2021
💔 -1 overall
This message was automatically generated. |
apurtell
commented
May 6, 2021
ContributorAuthor
Actual failing test |
Apache-HBase
commented
May 6, 2021
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
May 6, 2021
virajjasani
approved these changes
May 6, 2021
virajjasani
left a comment
Contributor
There was a problem hiding this comment.
So we are just removing what scheduled Procs are anyways going to do right, sounds good. +1
asfgit pushed a commit
that referenced
this pull request
May 6, 2021
…itor (#3234) In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Michael Stack <stack@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
asfgit pushed a commit
that referenced
this pull request
May 6, 2021
…itor (#3234) In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Michael Stack <stack@apache.org> Signed-off-by: Viraj Jasani <vjasani@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In CatalogJanitor we schedule GCRegionProcedure to clean up both filesystem and in-memory state after a split, and GCMultipleMergedRegionsProcedure to do the same for merges. Both of these procedures clean up in-memory state, but CatalogJanitor also does this redundantly just after scheduling the procedures. The cleanup should be done in only one place. Presumably we are using the procedures to do it in a principled way. This is least a nit, but probably a source of future bugs. Remove the redundancy in CatalogJanitor and fix any follow on issues, like test failures.