Uh oh!
There was an error while loading. Please reload this page.
HBASE-28215: region reopen procedure batching/throttling - #5534
Merged
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Apache-HBase
commented
Nov 22, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 22, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 22, 2023
💔 -1 overall
This message was automatically generated. |
rmdmattinglyforce-pushed
the
HBASE-28215
branch
from
November 22, 2023 16:19
84ef2ec to
2f30fecCompareApache-HBase
commented
Nov 22, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 22, 2023
💔 -1 overall
This message was automatically generated. |
rmdmattingly
commented
Nov 22, 2023
ContributorAuthor
I believe that test failure |
Apache-HBase
commented
Nov 22, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 22, 2023
🎊 +1 overall
This message was automatically generated. |
Uh oh!
There was an error while loading. Please reload this page.
Apache-HBase
commented
Nov 23, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 23, 2023
💔 -1 overall
This message was automatically generated. |
Apache9
reviewed
Nov 23, 2023
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Comment on lines
+173
to
+192
| // we need to create a set of region names because the HRegionLocation hashcode is only | ||
| // based | ||
| // on the server name | ||
| Set<byte[]> currentRegionBatchNames = currentRegionBatch.stream() | ||
| .map(r -> r.getRegion().getRegionName()).collect(Collectors.toSet()); | ||
| currentRegionBatch = regions.stream() | ||
| .filter(r -> currentRegionBatchNames.contains(r.getRegion().getRegionName())) | ||
| .collect(Collectors.toList()); | ||
| if (currentRegionBatch.isEmpty()) { | ||
| if (regions.isEmpty()) { | ||
| return Flow.NO_MORE_STATE; | ||
| } else { | ||
| setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS); | ||
| if (reopenBatchBackoffMillis > 0) { | ||
| backoff(reopenBatchBackoffMillis); | ||
| } | ||
| return Flow.HAS_MORE_STATE; | ||
| } | ||
| } | ||
| if (regions.stream().anyMatch(loc -> canSchedule(env, loc))) { | ||
| if (currentRegionBatch.stream().anyMatch(loc -> canSchedule(env, loc))) { |
Contributor
There was a problem hiding this comment.
I think the changes in this method are sort of confusing. I think the division of labor should be:
REOPEN_TABLE_REGIONS_REOPEN_REGIONS -- breaks the set of regions into a smaller batch, and reopens that batch. The code you have for that looks okREOPEN_TABLE_REGIONS_CONFIRM_REOPENED -- checks if any regions are still needing reopen. If so, go back to REOPEN_REGIONS. This is how it used to work.
If we totally reverted the changes in REOPEN_TABLE_REGIONS_CONFIRM_REOPENED, I think we'd have successfully broken the full region list into batches and the flow would work, just without backoff. So in this method we really just need to add the backoff.
The backoff as you have it below looks ok at first glance. This is just a long winded way of saying we should drop the highlighted code :)
ContributorAuthor
There was a problem hiding this comment.
I think this complexity is necessary in order to differentiate between are there more regions to reopen vs are regions still reopening in the current batch, the former being a state change and fixed backoff, and the latter being a retry and exponentially increasing backoff
Contributor
There was a problem hiding this comment.
Let's look back at the original impl of CONFIRM_REOPEN. It only does the backoff if there are still regions to reopen and none of them are schedulable. Meaning we back off in order to wait for regions to be schedulable.
Now let's look at your new code. I see what you are saying. You are basically doing:
- If currentRegionBatch are all reopened, exit or process the next batch
- Otherwise, can any of the remaining be scheduled? If not, exponential backoff. If so, process next batch.
The key thing there is that "process next batch". You'd think that would be "re-schedule existing batch". But as is, when you go to process next batch, the first thing REOPEN_REGIONS does is create a new currentRegionBatch. So lets say for the first batch of 50, only 10 reopen. With the current design, you'd expect it to try again to reopen those remaining 40. But it will actually reopen another 50 (40 of which are likely to have been in the last batch).
TBH I don't think there's a huge problem with that approach. But given that's how it works, we can simplify this code as I described before -- we don't really care about currentRegionBatch here, just that if any regions are left to reopen and schedulable.
If we wanted to do what I think you are intending to do, we probably need to complicate things a bit further. Maybe we update REOPEN_REGIONS to only create currentRegionBatch if it's currently null. Then in CONFIRM_REOPEN, null it out after confirming that they've been reopened. That way, in the above example when we kickoff REOPEN_REGIONS again we'll only open the remaining 40.
Does that make sense? Between these 2 options I might opt for the simpler one where we keep REOPEN_REGIONS as is and remove most of the diff from CONFIRM_REOPEN.
Contributor
There was a problem hiding this comment.
On second thought, after a discussion on https://issues.apache.org/jira/browse/HBASE-25549, i wonder if we should take the safer approach where require each batch to finish before scheduling more. This feature could serve a dual purpose of progressive rollout and rate limiting.
We could also trivially update this code to do an actual progressive deploy -- first batch size 1, then 2, then 4, etc up to the current batch size config you added. At that point it stays at that max concurrency until completion.
Thoughts on adding that? I think we just need one more field for currentBatchSize which we increment after each batch up to reopenBatchSize. It might make sense to persist this new field in the proto.
ContributorAuthor
There was a problem hiding this comment.
Now let's look at your new code. I see what you are saying. You are basically doing:
If currentRegionBatch are all reopened, exit or process the next batch
Otherwise, can any of the remaining be scheduled? If not, exponential backoff. If so, process next batch.
I think that's missing a few distinct outcomes of CONFIRM_REOPENED, and that's where this complexity comes into play. Here are the possibilities:
- The current batch has reopened successfully and there are no more regions to produce a next batch, so finish
- The current batch has reopened successfully and there are more regions to produce a next batch, so set the next state to
REOPEN_REGIONSand suspend. This will process the next batch - The current batch has not reopened successfully, and some of the current batch can still be scheduled. In this case return to
REOPEN_REGIONSto avoid an infinite loop due to a bypassed proc or something. This could technically reprocess the next batch, or redefine what the "current batch" looks like, but I think that's ok given the exceptional nature of this case - The current batch has not reopened successfully, and none of the current batch can still be scheduled (because they have already been scheduled). In this case we're just waiting on TRSPs to complete, so retain current state of
CONFIRM_REOPENEDand retry w/ increasing backoff
If I just revert this diff as described then this procedure will just get stuck in infinite loops (that I think could theoretically be exited if TRSP finishes quickly enough?) and all unit tests fail
We could also trivially update this code to do an actual progressive deploy -- first batch size 1, then 2, then 4, etc up to the current batch size config you added. At that point it stays at that max concurrency until completion.
I like this idea, and I think it would be trivial to add in the current state of this PR. We'd just need to store a maxReopenBatchSize and increase the reopenBatchSize to Math.min(2*reopenBatchSize, maxReopenBatchSize) each time we enter state 2 above
Contributor
There was a problem hiding this comment.
Ok lets table the missing outcomes/simplification discussion in that case, since we'll want to keep the current logic for the progressive reopen. We'll also want to do as i said, where in CONFIRM we update currentRegionBatch to filter out the reopened regions, and then in REOPEN we should only create a new currentRegionBatch if the existing isEmpty. That way if there's some problematic region in the batch, we won't move on to the next batch until its been reopened. Right now I think we'd just move on and create a new batch including the problematic region + more?
Once we have that in there, I need to take another look at the logic here. I think it could be a bit clearer, but don't want to make any suggestions until we have the complete impl in place.
In terms of progressive, it'd be nice to start with 1 region but otherwise agree with your Math.min idea.
ContributorAuthor
There was a problem hiding this comment.
Sounds good, I just pushed a change which does the following:
- each reopen proc will start with a batch size of 1
- the batch size will double after each confirmed batch (until we hit our max)
- our tests confirm the intended batching behavior, and that each region is only opened once
| retryCounter = null; | ||
| setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS); | ||
| if (reopenBatchBackoffMillis > 0) { | ||
| backoff(reopenBatchBackoffMillis); |
Contributor
There was a problem hiding this comment.
This is a little confusing to read. Specifically, the way to backoff is by throwing ProcedureSuspendedException but below you are returning Flow.HAS_MORE_STATE. It would be nice to indicate that the return only happens if we don't back-off. One way to do that would be to add a comment below. But it might be better to rename backoff to setBackoffState() and throw a ProcedureSuspendedException here. You're already throwing it below (which is redundant given backoff throws it now)
Contributor
There was a problem hiding this comment.
One other thing to note -- since we do skipPersistence(), we won't persist the setNextState change above. I think this should be fine. The implication is that in the case of a crash/restart while the procedure is suspended, we might backoff twice which is not the end of the world.
Apache-HBase
commented
Nov 28, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 28, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 28, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 29, 2023
🎊 +1 overall
This message was automatically generated. |
Comment on lines
+195
to
+199
| Set<byte[]> currentRegionBatchNames = currentRegionBatch.stream() | ||
| .map(r -> r.getRegion().getRegionName()).collect(Collectors.toSet()); | ||
| currentRegionBatch = regions.stream() | ||
| .filter(r -> currentRegionBatchNames.contains(r.getRegion().getRegionName())) | ||
| .collect(Collectors.toList()); |
Contributor
There was a problem hiding this comment.
Are you sure we need to do this? What if we did something like this:
caseREOPEN_TABLE_REGIONS_CONFIRM_REOPENED:
// update region lists based on what's been reopenedregions = filterReopened(regions);
currentRegionBatch = filterReopened(currentRegionBatch)
// existing batch didn't fully reopen, so try to resolve that first.// since this is a retry, don't do the batch backoffif (!currentRegionBatch.isEmpty()) {
returnreopenIfSchedulable(currentRegionBatch, false);
}
if (regions.isEmpty()) {
returnFlow.NO_MORE_STATE;
}
// no batching or batch is finished, schedule more from main regionsreturnreopenIfSchedulable(regions, true);
default:
thrownewUnsupportedOperationException("unhandled state=" + state);
}
...
privateList<HRegionLocation> filterReopened(List<HRegionLocation> regionsToCheck) {
returnregionsToCheck.stream().map(env.getAssignmentManager().getRegionStates()::checkReopened)
.filter(l -> l != null).collect(Collectors.toList());
}
privatevoidreopenIfSchedulable(List<HRegionLocations> regionsToReopen, booleanshouldBatchBackoff) throwsProcedureSuspendedException {
if (regionsToReopen.stream().anyMatch(loc -> canSchedule(env, loc))) {
retryCounter = null;
setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS);
if (shouldBatchBackoff && reopenBatchBackoffMillis > 0) {
reopenBatchSize = Math.min(reopenBatchSizeMax, 2 * reopenBatchSize);
setBackoffStateAndSuspend(reopenBatchBackoffMillis);
} else {
returnFlow.HAS_MORE_STATE;
}
}
// We can not schedule TRSP for all the regions need to reopen, wait for a while and retry// again.if (retryCounter == null) {
retryCounter = ProcedureUtil.createRetryCounter(env.getMasterConfiguration());
}
longbackoffMillis = retryCounter.getBackoffTimeAndIncrementAttempts();
LOG.info(
"There are still {} region(s) which need to be reopened for table {}. {} are in "
+ "OPENING state, suspend {}secs and try again later",
regions.size(), tableName, regionsToCheck.size(), backoffMillis / 1000);
setBackoffStateAndSuspend(backoffMillis);
}It feels cleaner to read and understand what's happening. As mentioned in another comment, I'd recommend updating REOPEN_REGIONS to:
// if we didn't finish reopening the last batch yet, let's keep trying until we do.// at that point, the batch will be empty and we can generate a new batchif (currentRegionBatch.isEmpty()) {
currentRegionBatch = regions.stream().limit(reopenBatchSize).collect(Collectors.toList());
}ContributorAuthor
There was a problem hiding this comment.
Agreed all around about the feedback here, just pushed a simplification that follows this advice almost exactly. The tests are all still passing of course
Apache-HBase
commented
Nov 29, 2023
🎊 +1 overall
This message was automatically generated. |
rmdmattinglyforce-pushed
the
HBASE-28215
branch
from
November 29, 2023 20:15
f7c146f to
afa02f9Compare| if (regionsToReopen.stream().anyMatch(loc -> canSchedule(env, loc))) { | ||
| retryCounter = null; | ||
| setNextState(ReopenTableRegionsState.REOPEN_TABLE_REGIONS_REOPEN_REGIONS); | ||
| reopenBatchSize = Math.min(reopenBatchSizeMax, 2 * reopenBatchSize); |
Contributor
There was a problem hiding this comment.
i think we should only increase the batch size in the below if block
ContributorAuthor
There was a problem hiding this comment.
That would mean that the default case of reopenBatchBackoffMillis = 0 is stuck processing single region batches indefinitely
Contributor
There was a problem hiding this comment.
oh wait, i dont think we want progressive reopen enabled by default right away. i was actually thinking that this progressive batching would only apply if one of the new configs is enabled. i missed that above.
we might want to change our default values so that we can tell if a user actually changed the configs, and only enable progressive if so.
ContributorAuthor
There was a problem hiding this comment.
oh interesting, ok that makes sense
Comment on lines
+57
to
+62
| public static final String REOPEN_BATCH_BACKOFF_MILLIS_KEY = | ||
| "hbase.table.regions.reopen.batch.backoff.ms"; | ||
| public static final long REOPEN_BATCH_BACKOFF_MILLIS_DEFAULT = 0L; | ||
| public static final String REOPEN_BATCH_SIZE_MAX_KEY = | ||
| "hbase.table.regions.reopen.batch.size.max"; | ||
| public static final int REOPEN_BATCH_SIZE_MAX_DEFAULT = Integer.MAX_VALUE; |
Contributor
There was a problem hiding this comment.
nitpick: we might want to have these names be hbase.reopen.table.regions.progressive.batch.size.max and backoff.ms. It's more of a progressive reopen system now, and i reordered the table.regions.reopen to align more with the procedure name.
Uh oh!
There was an error while loading. Please reload this page.
bbeaudreault
approved these changes
Dec 1, 2023
bbeaudreault
left a comment
Contributor
There was a problem hiding this comment.
LGTM! @Apache9 anything else on your end?
Apache-HBase
commented
Dec 2, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 2, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 2, 2023
💔 -1 overall
This message was automatically generated. |
bbeaudreault
commented
Dec 2, 2023
Contributor
No rush, but looks like test failures are related and new compile/javac warnings |
rmdmattingly
commented
Dec 2, 2023
ContributorAuthor
The javac warning is legit, will get to that asap. The test failure was for the previous commit I believe |
Apache-HBase
commented
Dec 2, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 2, 2023
🎊 +1 overall
This message was automatically generated. |
rmdmattinglyforce-pushed
the
HBASE-28215
branch
from
December 3, 2023 15:32
eb51c02 to
8eec7ceCompareApache-HBase
commented
Dec 3, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 3, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 3, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 3, 2023
💔 -1 overall
This message was automatically generated. |
rmdmattinglyforce-pushed
the
HBASE-28215
branch
from
December 3, 2023 19:59
8eec7ce to
0d010a0CompareApache-HBase
commented
Dec 3, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 3, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 4, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 4, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 4, 2023
🎊 +1 overall
This message was automatically generated. |
bbeaudreault
approved these changes
Dec 4, 2023
bbeaudreault pushed a commit
that referenced
this pull request
Dec 4, 2023
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit
that referenced
this pull request
Dec 4, 2023
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit
that referenced
this pull request
Dec 4, 2023
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit
to HubSpot/hbase
that referenced
this pull request
Dec 4, 2023
…tling (apache#5534) Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
kadirozde pushed a commit
to kadirozde/hbase
that referenced
this pull request
Jan 5, 2024
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-28215
The mass reopening of regions caused by a table descriptor modification can be quite disruptive. For latency/error sensitive workloads, like our user facing traffic, we need to be very careful about when we modify table descriptors, and it can be virtually impossible to do it painlessly for busy tables.
This PR introduces two new configurations:
@bbeaudreault@hgromer@eab148@bozzkar