Uh oh!
There was an error while loading. Please reload this page.
HBASE-27483 Expose table and region storefiles accessed days and size to the metrics - #4891
Open
2005hithlj wants to merge 1 commit into
Open
HBASE-27483 Expose table and region storefiles accessed days and size to the metrics#48912005hithlj wants to merge 1 commit into
2005hithlj wants to merge 1 commit into
Conversation
Apache-HBase
commented
Nov 21, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 21, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 21, 2022
🎊 +1 overall
This message was automatically generated. |
taklwu
reviewed
Nov 21, 2022
taklwu
left a comment
Contributor
There was a problem hiding this comment.
this is an interesting change, very nice.
can you fix the spotbug error ?
| public static long ONE_DAY_MS = 24 * 3600 * 1000; | ||
| public static final String STOREFILES_ACCESSED_DAYS_THRESHOLDS = | ||
| "hbase.region.storefiles.accessed.days.thresholds"; | ||
| public static final int[] STOREFILES_ACCESSED_DAYS_THRESHOLDS_DEFAULT = { 7, 30, 90 }; |
Contributor
There was a problem hiding this comment.
can we add few line of javadoc or some line in README.md how to use this ?
Contributor
There was a problem hiding this comment.
One more question, if we are using cache / bucket cache heavily, will this metric retain at the time the store file flushes or opened for cache? It’s fine as is without caring the cache , just want to learn more here
ContributorAuthor
There was a problem hiding this comment.
Yeah, in this situation, the metrics will retain at the time the store file flushes or opened for cache, although the cached data of store file is accessed at a new time. I'm not sure whether it is critical issue.
| @Override | ||
| public void run() { | ||
| Map<TableName, MetricsTableValues> localMetricsTableMap = new HashMap<>(); | ||
Contributor
There was a problem hiding this comment.
[nit] is this added by spotless?
Suggested change
Comment on lines
+440
to
+442
| metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed7DaysSize", 0, agg); | ||
| metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed30DaysSize", 0, agg); | ||
| metricsHelper.assertGaugeGt(prefix + "storeFilesAccessed90DaysSize", 0, agg); |
Contributor
There was a problem hiding this comment.
[nit] is there any way to verify non-zero metric for storeFilesAccessed%sDaysSize in test, e.g. if we can mock the access time ?
2005hithlj
commented
Nov 22, 2022
ContributorAuthor
@taklwu sir. Thanks for your review. Please wait for my new commit. |
2005hithlj
commented
Nov 30, 2022
ContributorAuthor
@taklwu sir, I'm sorry to have kept you waiting so long. |
Apache-HBase
commented
Nov 30, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 30, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Nov 30, 2022
🎊 +1 overall
This message was automatically generated. |
2005hithlj
commented
Dec 9, 2022
ContributorAuthor
@taklwu Could you take a look? Thanks. |
Apache9
reviewed
Dec 22, 2022
| String ROW_READS_ONLY_ON_MEMSTORE_DESC = "Row reads happening completely out of memstore"; | ||
| String MIXED_ROW_READS = "mixedRowReadsCount"; | ||
| String MIXED_ROW_READS_ON_STORE_DESC = "Row reads happening out of files and memstore on store"; | ||
| String STOREFILES_ACCESSED_DAYS_AND_SIZE = "storeFilesAccessed%sDaysSize"; |
| Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = | ||
| store.getStoreFilesAccessTimeAndSize(); | ||
| Long now = EnvironmentEdgeManager.currentTime(); |
Contributor
There was a problem hiding this comment.
Use long instead of Long as later we will do calculation?
ContributorAuthor
There was a problem hiding this comment.
Use
longinstead ofLongas later we will do calculation?
OK, sir.
Uh oh!
There was an error while loading. Please reload this page.
Apache-HBase
commented
Dec 22, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 22, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 22, 2022
💔 -1 overall
This message was automatically generated. |
2005hithlj
commented
Dec 23, 2022
ContributorAuthor
The failed UTs seems to have nothing to do with my modification. |
2005hithlj
commented
Dec 23, 2022
ContributorAuthor
@Apache9 sir. Could you take a look? Thanks. |
Apache9
commented
Apr 8, 2023
Contributor
Please fix the conflicts. And how do you plan to reduce the pressure on NN when fetching the accessTime? Or is this not a big problem? Thanks. |
Apache-HBase
commented
May 11, 2023
🎊 +1 overall
This message was automatically generated. |
2005hithlj
commented
May 11, 2023
ContributorAuthor
@Apache9 sir. I have a new commit. Could you take a look? Thanks. |
Apache-HBase
commented
May 11, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 11, 2023
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
May 15, 2023
| mt.perStoreMixedReadCount.put(tempKey, mixedReadCount); | ||
| Map<String, Pair<Long, Long>> sfAccessTimeAndSizeMap = | ||
| store.getStoreFilesAccessTimeAndSize(); | ||
| Long now = EnvironmentEdgeManager.currentTime(); |
Contributor
There was a problem hiding this comment.
Just use long instead of Long?
| Long size = pair.getSecond(); | ||
| for (int threshold : storeFilesAccessedDaysThresholds) { | ||
| Long sumSize = mt.storeFilesAccessedDaysAndSize.get(threshold); | ||
| if ((now - accessTime) >= threshold * ONE_DAY_MS) { |
Contributor
There was a problem hiding this comment.
So here if the accessTime is very old, we will increase the sumSize for all the thresholds? Is this by design? Seems a bit strange...
ContributorAuthor
There was a problem hiding this comment.
@Apache9 sir.
Yes, this is by design. For instance, if an HFile has not been accessed for 90 days, it is certain that it has also not been accessed for 30 and 7 days.
And what is your suggestion?
Contributor
There was a problem hiding this comment.
OK, so the metric here is 'the number if store files which are not accessed by at least xx days'? Then I think we;d better have a better name for it? Looking at 'storeFilesAccessedDays', I can not get the meaning unless you explain it...
Apache-HBase
commented
May 16, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 16, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 16, 2023
🎊 +1 overall
This message was automatically generated. |
Apache9
commented
Aug 31, 2025
Contributor
Any updates here? @2005hithlj Thanks. |
2005hithlj
commented
Sep 4, 2025
ContributorAuthor
@Apache9 Sorry sir, I've been a bit busy at work lately. I don't have extra bandwidth to do anything else at the moment. |
Apache9
commented
Sep 4, 2025
Contributor
OK, then we can move it out of 3.0.0 first. Can come back later when you have time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-27483