Uh oh!
There was an error while loading. Please reload this page.
HBASE-28687 BackupSystemTable#checkSystemTable should ensure system tables are enabled - #6018
Merged
Conversation
Apache-HBase
commented
Jun 21, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2024
💔 -1 overall
This message was automatically generated. |
ContributorAuthor
Hmm. Test failures could be related. Looking into it edit: they pass on my machine 😓 |
Apache-HBase
commented
Jun 21, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2024
💔 -1 overall
This message was automatically generated. |
ContributorAuthor
This same test is failing across multiple PRs: #6017 |
Apache-HBase
commented
Jun 21, 2024
💔 -1 overall
This message was automatically generated. |
ContributorAuthor
Okay, I can reproduce the failures if I run all backups tests locally rather than just the one. I bet this has something to do with all of the inheritance going on in these unit tests |
bbeaudreault
approved these changes
Jun 21, 2024
bbeaudreault
left a comment
Contributor
There was a problem hiding this comment.
IMO enabling them makes sense since kicking off a backup is an implicitly request to use the backup state. It's also consistent with the other states handled here.
The only other thing I'd say is, do we need to handle exceptions? For example if we check that the table is disabled and it is, so we go to enable and it's already enabled. I've seen this problem in other areas where we handle table lifecycle automatically. Basically if multiple callers conflict, it throws an exception that could easily have been handled.
If this is already protected by the backup lock, then may not be necessary.
Comment on lines
+121
to
+151
| @Before | ||
| public void ensurePreviousBackupTestsAreCleanedUp() throws Exception { | ||
| // Every operation here may not be necessary for any given test, | ||
| // some often being no-ops. the goal is to help ensure atomicity | ||
| // of that tests that implement TestBackupBase | ||
| try (BackupAdmin backupAdmin = getBackupAdmin()) { | ||
| backupManager.finishBackupSession(); | ||
| backupAdmin.listBackupSets().forEach(backupSet -> { | ||
| try { | ||
| backupAdmin.deleteBackupSet(backupSet.getName()); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| } catch (Exception ignored) { | ||
| } | ||
| Arrays.stream(TEST_UTIL.getAdmin().listTableNames()) | ||
| .filter(tableName -> !tableName.isSystemTable()).forEach(tableName -> { | ||
| try { | ||
| TEST_UTIL.truncateTable(tableName); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| TEST_UTIL.getMiniHBaseCluster().getRegionServerThreads().forEach(rst -> { | ||
| try { | ||
| LogRoller walRoller = rst.getRegionServer().getWalRoller(); | ||
| walRoller.requestRollAll(); | ||
| walRoller.waitUntilWalRollFinished(); | ||
| } catch (Exception ignored) { | ||
| } | ||
| }); | ||
| } |
ContributorAuthor
There was a problem hiding this comment.
After some playing around, this seemed like a pretty good way to get TestBackupMerge, and others, to consistently pass even when ran in bulk. Another option would be to get rid of the TestBackupMerge inheritance, but that would be a really annoying change to implement, and would probably be significantly slower in the build pipeline
Member
There was a problem hiding this comment.
Sure, let's try it.
For what it's worth, I've had success with refactoring inheritance out of these junit4 tests by moving shared logic into an ExternalResource implementation. It doesn't let you reuse @Test implementations, but it does make implementing before/after logic more composable.
rmdmattingly
commented
Jun 23, 2024
ContributorAuthor
Agreed with the feedback, I added handling of TableNotDisabled exceptions |
Apache-HBase
commented
Jun 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 23, 2024
🎊 +1 overall
This message was automatically generated. |
ndimiduk
approved these changes
Jun 26, 2024
Comment on lines
+121
to
+151
| @Before | ||
| public void ensurePreviousBackupTestsAreCleanedUp() throws Exception { | ||
| // Every operation here may not be necessary for any given test, | ||
| // some often being no-ops. the goal is to help ensure atomicity | ||
| // of that tests that implement TestBackupBase | ||
| try (BackupAdmin backupAdmin = getBackupAdmin()) { | ||
| backupManager.finishBackupSession(); | ||
| backupAdmin.listBackupSets().forEach(backupSet -> { | ||
| try { | ||
| backupAdmin.deleteBackupSet(backupSet.getName()); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| } catch (Exception ignored) { | ||
| } | ||
| Arrays.stream(TEST_UTIL.getAdmin().listTableNames()) | ||
| .filter(tableName -> !tableName.isSystemTable()).forEach(tableName -> { | ||
| try { | ||
| TEST_UTIL.truncateTable(tableName); | ||
| } catch (IOException ignored) { | ||
| } | ||
| }); | ||
| TEST_UTIL.getMiniHBaseCluster().getRegionServerThreads().forEach(rst -> { | ||
| try { | ||
| LogRoller walRoller = rst.getRegionServer().getWalRoller(); | ||
| walRoller.requestRollAll(); | ||
| walRoller.waitUntilWalRollFinished(); | ||
| } catch (Exception ignored) { | ||
| } | ||
| }); | ||
| } |
Member
There was a problem hiding this comment.
Sure, let's try it.
For what it's worth, I've had success with refactoring inheritance out of these junit4 tests by moving shared logic into an ExternalResource implementation. It doesn't let you reuse @Test implementations, but it does make implementing before/after logic more composable.
ndimiduk pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 26, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 26, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit
to ndimiduk/hbase
that referenced
this pull request
Jun 26, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit
that referenced
this pull request
Jun 27, 2024
…ables are enabled (#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit
that referenced
this pull request
Jun 27, 2024
…ables are enabled (#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
ndimiduk pushed a commit
that referenced
this pull request
Jun 27, 2024
…ables are enabled (#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly added a commit
to HubSpot/hbase
that referenced
this pull request
Jun 28, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly added a commit
to HubSpot/hbase
that referenced
this pull request
Jun 28, 2024
…ables are enabled (apache#6018) Co-authored-by: Ray Mattingly <rmattingly@hubspot.com> Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
rmdmattingly added a commit
to HubSpot/hbase
that referenced
this pull request
Jul 8, 2024
…ld ensure system tables are enabled (apache#6018) (#102) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
rmdmattingly added a commit
to HubSpot/hbase
that referenced
this pull request
Jul 8, 2024
…ld ensure system tables are enabled (apache#6018) (#103) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See https://issues.apache.org/jira/browse/HBASE-28687
If the backup system tables become disabled, then we enter a state which the backup client will not recover from. Without manual intervention, every subsequent backup attempt will fail on BackupSystemTable's calls to waitForSystemTable.
This checkSystemTable method already ensures that the tables exist — it should also ensure that the tables are enabled before we await that condition.
Alternatively, we could fast-fail if the tables are disabled rather than awaiting an enabled state that will never occur.
cc @ndimiduk@charlesconnell@hgromer