Uh oh!
There was an error while loading. Please reload this page.
HBASE-28399 region size can be wrong from RegionSizeCalculator - #5700
Open
frostruan wants to merge 2 commits into
Open
HBASE-28399 region size can be wrong from RegionSizeCalculator#5700frostruan wants to merge 2 commits into
frostruan wants to merge 2 commits into
Conversation
Apache-HBase
commented
Feb 23, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 23, 2024
💔 -1 overall
This message was automatically generated. |
frostruan
commented
Feb 23, 2024
ContributorAuthor
Some unit tests failed. Let me try to fix them. |
Apache-HBase
commented
Feb 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 23, 2024
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
Feb 26, 2024
| byte[] regionId = regionLoad.getRegionName(); | ||
| long regionSizeBytes = | ||
| ((long) regionLoad.getStoreFileSize().get(Size.Unit.MEGABYTE)) * MEGABYTE; | ||
| long regionSizeBytes = (long) regionLoad.getMemStoreSize().get(Size.Unit.BYTE) |
Contributor
There was a problem hiding this comment.
Why we use Unit.MEGABYTE and then multiply 1024 * 1024 in the past? Can you find any related issues about this? Seems strange...
Contributor
There was a problem hiding this comment.
Seems it was like this in the first place...
ContributorAuthor
There was a problem hiding this comment.
Thanks for reviewing Duo.
From my understanding, before we introduce the Size api, we got region size by getMemstoreSizeMB and getStoreFileSizeMB, so I guess this is just to keep the same when applied the new Size api ?
Contributor
There was a problem hiding this comment.
OK, checked the code on how we constructor the store file size and memstore size, the default unit is MB, so it is useless to pass an unit less than MB here...
Contributor
There was a problem hiding this comment.
But anyway, I think the problem here is we also need to account memstore size when calculating region size?
ContributorAuthor
There was a problem hiding this comment.
Yes. otherwise the data in memstore will be lost.
Member
There was a problem hiding this comment.
I think that Hadoop assumes these split sizes are in megabytes and so we follow suit.
To protect against loss of precision, when the bytes-unit value is non-0, we can apply a minimum of 1mb. We should always add this minimum when running against an online-cluster.
When running against a snapshot, I'm not sure. MR over snapshots instantiates the region in the mapper process -- I assume that also reads the WAL and populates a memstore. In that case, we need the 1mb minimum here too. If not, we can permit the 0 to pass through and give the empty split optomization a chance.
ContributorAuthor
There was a problem hiding this comment.
To protect against loss of precision, when the bytes-unit value is non-0, we can apply a minimum of 1mb. We should always add this minimum when running against an online-cluster.
Agree. And I believe this problem has been solved in https://issues.apache.org/jira/browse/HBASE-26609
When running against a snapshot, I'm not sure. MR over snapshots instantiates the region in the mapper process -- I assume that also reads the WAL and populates a memstore. In that case, we need the 1mb minimum here too. If not, we can permit the 0 to pass through and give the empty split optomization a chance.
The value of snapshot input split length will always be 0.
https://github.com/apache/hbase/blob/rel/3.0.0-beta-1/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java#L185
I think maybe this should be increased to 1MB too ?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.