Uh oh!
There was an error while loading. Please reload this page.
HBASE-22285 A normalizer which merges small size regions with adjacen… - #931
Closed
mnpoonia wants to merge 8 commits into
Closed
HBASE-22285 A normalizer which merges small size regions with adjacen…#931mnpoonia wants to merge 8 commits into
mnpoonia wants to merge 8 commits into
Conversation
Apache-HBase
commented
Dec 12, 2019
💔 -1 overall
This message was automatically generated. |
| mergeEnabled = masterRpcServices.isSplitOrMergeEnabled(null, | ||
| RequestConverter.buildIsSplitOrMergeEnabledRequest(Admin.MasterSwitchType.MERGE)).getEnabled(); | ||
| } catch (ServiceException se) { | ||
| LOG.debug("Unable to determine whether merge is enabled", se); |
ContributorAuthor
There was a problem hiding this comment.
I think we can make it WARN but not error. Its a valid scenario for table to disable merge and still at cluster level have normalizer enabled.
Uh oh!
There was an error while loading. Please reload this page.
| * <li> get all regions of a given table | ||
| * <li> get avg size S of each region (by total size of store files reported in RegionLoad) | ||
| * <li> Otherwise, two region R1 and its smallest neighbor R2 are merged, | ||
| * if R1 + R1 < S, and all such regions are returned to be merged |
ContributorAuthor
There was a problem hiding this comment.
thanks for catching that
| candidateIdx++; | ||
| } | ||
| } catch (Exception e) { | ||
| e.printStackTrace(); |
Contributor
There was a problem hiding this comment.
nit: LOG.error("MergeNormalizer failed for region {} and {}", r1Name, r2Name, e);
| "Table " + table + ", small region size: " + regionSize + " plus its neighbor size: " | ||
| + regionSize2 + ", less than the avg size " + avgRegionSize + ", merging them"); | ||
| plans.add(new MergeNormalizationPlan(hri, hri2)); | ||
| candidateIdx++; |
Contributor
There was a problem hiding this comment.
should be candidateIdx = candidateIdx + 2? e.g. if we have 3 regions: r1, r2, r3 and if we consider r1, r2 for merging, plans should have only one entry with Plan(r1, r2) right? but with candidateIdx++, we might end up adding 2 plans: Plan(r1, r2) and Plan(r2, r3)?
ContributorAuthor
There was a problem hiding this comment.
There is candidateIdx++ outside if block and was kept outside purposefully.
Uh oh!
There was an error while loading. Please reload this page.
wchevreuil
left a comment
Contributor
There was a problem hiding this comment.
Additional to @virajjasani observations, had made some minor suggestions over the code. On Another thought, the main structure for computePlanForTable looks very similar to the one from SimpleRegionNormalizer. Would it be too difficult to refactor _SimpleRegionNormalizer so that both normalizer implementations could make most of code reuse?
| } | ||
| HRegionInfo hri2 = tableRegions.get(candidateIdx + 1); | ||
| long regionSize2 = getRegionSize(hri2); | ||
| if (regionSize >= 0 && regionSize2 >= 0 && regionSize + regionSize2 < avgRegionSize) { |
Contributor
There was a problem hiding this comment.
Maybe worth logging (debug?) regions that didn't pass this check.
Uh oh!
There was an error while loading. Please reload this page.
Comment on lines
+119
to
+123
| if (!(new Timestamp(hriTime.getTime() + TimeUnit.DAYS.toMillis(MIN_DURATION_FOR_MERGE))) | ||
| .after(currentTime) | ||
| || !(new Timestamp( | ||
| hri2Time.getTime() + TimeUnit.DAYS.toMillis(MIN_DURATION_FOR_MERGE))) | ||
| .after(currentTime)) { |
Contributor
There was a problem hiding this comment.
Few minor suggestions:
- Maybe worth mention this extra condition on the class javadoc comment.
- The checking logic seems duplicate. Place the check into a separate method:
!(new Timestamp(hriTime.getTime() + TimeUnit.DAYS.toMillis(MIN_DURATION_FOR_MERGE))) .after(currentTime) - Make MIN_DURATION_FOR_MERGE configurable?
Apache-HBase
commented
Dec 17, 2019
💔 -1 overall
This message was automatically generated. |
ContributorAuthor
After giving it a thought i believe we should have two normalizers one which splits and second which merges and may be third SimpleRegionNormalizer which uses these two and together. |
mnpoonia
commented
Dec 17, 2019
ContributorAuthor
Also i will make the MIN_DURATION_FOR_MERGE configurable. |
wchevreuil
commented
Dec 18, 2019
Contributor
I'm not sure moving logic to an util class is a good design practice. Here I think we could probably do a strategy pattern: An abstract normalizer class implementing most of computePlanForTable logic, with only the parts related to picking a region to the region plan and what actions should be perfomed over the region plan (merge, or split, or both) being delegated to methods that would be implemented differently on each of the subclasses. |
Apache-HBase
commented
Dec 19, 2019
💔 -1 overall
This message was automatically generated. |
| private static final int MIN_DURATION_FOR_MERGE = 2; | ||
| @Override | ||
| public List<NormalizationPlan> computePlanForTable(TableName table) throws HBaseIOException { |
Contributor
There was a problem hiding this comment.
Could we implement all common parts of computePlanForTable in the base parent class, then only merge or split normalizer specifc logic would need to go on the implementations?
Comment on lines
+77
to
+84
| if (isOldEnoughToMerge(hri) || isOldEnoughToMerge(hri2)) { | ||
| LOG.info( | ||
| "Table {}, small region size: {} plus its neighbor size: {}, less than the avg size " | ||
| + "{}, merging them", | ||
| table, regionSize, regionSize2, avgRegionSize); | ||
| plans.add(new MergeNormalizationPlan(hri, hri2)); | ||
| candidateIdx++; | ||
| } |
Contributor
There was a problem hiding this comment.
For example, this could go to a separate, abstract method computeNormalization in the base parent class, then MergeNormalizer would implement it as it is now, and SimpleRegionNormalizer would do it the way it does now.
…r for common functinality
Apache-HBase
commented
Dec 28, 2019
💔 -1 overall
This message was automatically generated. |
| RequestConverter.buildIsSplitOrMergeEnabledRequest(Admin.MasterSwitchType.MERGE)) | ||
| .getEnabled(); | ||
| } catch (ServiceException se) { | ||
| LOG.debug("Unable to determine whether merge is enabled", se); |
Contributor
There was a problem hiding this comment.
want to convert this to error or warn log?
| RequestConverter.buildIsSplitOrMergeEnabledRequest(Admin.MasterSwitchType.SPLIT)) | ||
| .getEnabled(); | ||
| } catch (ServiceException se) { | ||
| LOG.debug("Unable to determine whether merge is enabled", se); |
| private static final Log LOG = LogFactory.getLog(SimpleRegionNormalizer.class); | ||
| public class SimpleRegionNormalizer extends BaseNormalizer { | ||
| private static final Logger LOG = LoggerFactory.getLogger(SimpleRegionNormalizer.class); | ||
| private static final int MIN_REGION_COUNT = 3; |
Contributor
There was a problem hiding this comment.
MergeNormalizer and SimpleRegionNormalizer both can use same constant MIN_REGION_COUNT, may be good to move it to HConstant
ContributorAuthor
There was a problem hiding this comment.
I am not sure about this yet. Let me think a bit. The point is that both normalizer can have different requirement for the MIN_REGION_COUNT. But then we can't have both normalizer enabled together.
Contributor
There was a problem hiding this comment.
Right, also this is mostly same constant. if we want to use configs for these values(which we don't want I believe), then we could use it separately but here, may be we can refer to same constant.
Apache-HBase
commented
Dec 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 29, 2019
💔 -1 overall
This message was automatically generated. |
| int candidateIdx = 0; | ||
| while (candidateIdx < tableRegions.size()) { | ||
| if (candidateIdx == tableRegions.size() - 1) { |
Contributor
There was a problem hiding this comment.
we can get rid of this if condition with while( candidateIdx < tableRegions.size()-1 )
| plans.add(new MergeNormalizationPlan(hri, hri2)); | ||
| candidateIdx++; | ||
| } else { | ||
| LOG.debug("Skipping region {} of table {} with size {}", hri.getRegionId(), table, |
Contributor
There was a problem hiding this comment.
Instead of regionId, may be good to print region name with hri.getRegionNameAsString()
| if (splitEnabled) { | ||
| List<NormalizationPlan> splitNormalizationPlan = getSplitNormalizationPlan(table); | ||
| if (splitNormalizationPlan != null) { | ||
| plans = splitNormalizationPlan; |
Contributor
There was a problem hiding this comment.
To keep it aligned with mergeNormalization, here also, it's good to add all: plans.addAll(splitNormalizationPlan)
| } | ||
| } | ||
| if (mergeEnabled) { | ||
| List<NormalizationPlan> normalizationPlans = getMergeNormalizationPlan(table); |
Contributor
There was a problem hiding this comment.
nit: rename to mergeNormalizationPlans?
| private boolean shouldNormalize(TableName table) { | ||
| boolean normalize = false; | ||
| if (table == null || table.isSystemTable()) { | ||
| LOG.debug("Normalization of system table {} isn't allowed", table); |
Contributor
There was a problem hiding this comment.
I don't think table could be null here, and we can remove null check from this if condition. Even if null was possible, log message would not be appropriate: Normalization of system table null isn't allowed
Apache-HBase
commented
May 23, 2020
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…t regions