Uh oh!
There was an error while loading. Please reload this page.
HBASE-28465 Implementation of framework for time-based priority bucket-cache - #5793
Merged
Merged
Conversation
Apache-HBase
commented
Apr 4, 2024
💔 -1 overall
This message was automatically generated. |
wchevreuil
left a comment
Contributor
There was a problem hiding this comment.
Overall, looks good, I just think we need to add javadoc on public methods and explain properly when we will consider a file/block as cold/hot, specially when dealing with default tiering type.
| long diff = currentTimestamp - maxTimestamp.getAsLong(); | ||
| return diff <= hotDataAge; | ||
| } | ||
| return false; |
Contributor
There was a problem hiding this comment.
Shouldn't we return "true" by default? I.e., if not using DataTieringType.TIME_RANGE, this should consider all data as hot and let the LFU logic decide on the eviction right?
ContributorAuthor
There was a problem hiding this comment.
The logic reaches that point only when the DataTieringType is set to NONE, indicating that data tiering is disabled. If data tiering is disabled, we should not consider that data as special by marking it as hot, correct? Instead, it should be considered as cold.
For example, if I set it to true, it means we are indicating that data is hot even when data tiering is disabled.
Contributor
There was a problem hiding this comment.
For example, if I set it to true, it means we are indicating that data is hot even when data tiering is disabled.
Ok, I think there's a misunderstanding of concepts. Today, without this DataTiering thing, the behaviour is to cache everything and evict based on LFU (in other words, everything is hot). If I don't turn on DataTieringType.TIME_RANGE, I want things to keep working as today (cache everything and evict based on LFU). Maybe we can change the default name from NONE to ALL.
ContributorAuthor
There was a problem hiding this comment.
Even with this implementation, it will function the same. For example, in eviction, we'll prioritize evicting the cold data files first, which are all the files that satisfy the condition (isDataTieringEnabled(file) && !isHotData(file)). Once I identify these files, I will proceed to evict them and then hand over control to the rest of the logic.
ContributorAuthor
There was a problem hiding this comment.
Okay, I get your point. then we can eliminate isDataTieringEnabled methods, right? since we are considering everything as hot by default.
| HStoreFile hStoreFile = getHStoreFile(hFilePath); | ||
| if (hStoreFile == null) { | ||
| throw new DataTieringException( |
Contributor
There was a problem hiding this comment.
Rather than throwing exception, maybe just return false? I wonder if it would be possible that a compaction completed and moved the given file away just after we entered this method.
ContributorAuthor
There was a problem hiding this comment.
Wouldn't it be better to throw the exception and let the caller decide what action to take based on the context?
Contributor
There was a problem hiding this comment.
I'm thinking in terms of responsibilities and cohesion. Who should know how to classify a block as hot or cold for a file that is not in the list of regions stores? Shouldn't this be done here?
ContributorAuthor
There was a problem hiding this comment.
Got it. should we change in other places as well?
Uh oh!
There was an error while loading. Please reload this page.
| try { | ||
| Set<String> coldFilePaths = dataTieringManager.getColdDataFiles(allCachedBlocks); | ||
| assertEquals(1, coldFilePaths.size()); |
Contributor
There was a problem hiding this comment.
We should explain better why we only get 1 here, even though we say there are two non-hot files in the javadoc header.
Also, shouldn't we assert on the exact cold file, just to make sure our logic does mark the file we know is cold?
ContributorAuthor
There was a problem hiding this comment.
Yeah, good idea. Maybe I write another test to stress on cold data
vinayakphegde
commented
Apr 4, 2024
ContributorAuthor
Sure, We'll add that. |
Apache-HBase
commented
Apr 4, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 4, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 5, 2024
💔 -1 overall
This message was automatically generated. |
| if (exception != null) { | ||
| fail("Expected DataTieringException to be thrown"); | ||
| } | ||
| assertEquals(value, expectedResult); |
Contributor
There was a problem hiding this comment.
nit: assertEquals(expectedResult, value);
| if (exception != null) { | ||
| fail("Expected DataTieringException to be thrown"); | ||
| } | ||
| assertEquals(value, expectedResult); |
Contributor
There was a problem hiding this comment.
nit: assertEquals(expectedResult, value);
Apache-HBase
commented
Apr 5, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 5, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 5, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 6, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 6, 2024
🎊 +1 overall
This message was automatically generated. |
wchevreuil
commented
Apr 8, 2024
Contributor
LTGM, can we just address latest spotless failure? |
vinayakphegde
commented
Apr 8, 2024
ContributorAuthor
That's because of the Javadoc in the |
Apache-HBase
commented
Apr 8, 2024
🎊 +1 overall
This message was automatically generated. |
wchevreuil
approved these changes
Apr 8, 2024
jhungund
reviewed
Apr 8, 2024
| if (dataTieringType.equals(DataTieringType.TIME_RANGE)) { | ||
| long hotDataAge = getDataTieringHotDataAge(configuration); | ||
| HStoreFile hStoreFile = getHStoreFile(hFilePath); |
Contributor
There was a problem hiding this comment.
getConfiguratin(hFilePath) already traverses through the regions to get to the HStore. We will do it again within getHStoreFile. We can slightly improve by avoiding duplicate traversal to get HStore.
One Option I can think is to getStore first.
HStore store = getStore(hFilePath); <= We traverse only once.
Then,
Configuration configuration = getConfiguration(store);
HStoreFile file = getHStoreFile(store);
With this, we only have regions traversal only once.
ContributorAuthor
There was a problem hiding this comment.
No, it doesn't traverse through the regions. these are just map lookups.
Apache-HBase
commented
Apr 8, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 8, 2024
💔 -1 overall
This message was automatically generated. |
vinayakphegde
commented
Apr 8, 2024
ContributorAuthor
@wchevreuil, it seems like most of the tests have passed, and any failures were due to flaky tests. |
wchevreuil pushed a commit
that referenced
this pull request
Apr 12, 2024
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Apr 22, 2024
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Apr 25, 2024
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
May 20, 2024
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
vinayakphegde added a commit
to vinayakphegde/hbase
that referenced
this pull request
May 21, 2024
…t-cache (apache#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
May 21, 2024
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
jhungund pushed a commit
to janardhanrh/hbase
that referenced
this pull request
May 29, 2024
…t-cache (apache#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: I22072f951ddfa8768e820952f63bbc2aa3cd5217
wchevreuil pushed a commit
that referenced
this pull request
Jun 24, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Jul 10, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Jul 11, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Aug 5, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Sep 1, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Sep 1, 2025
…t-cache (apache#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: If213337be959a392a9bc55aba63b4d033df8e729
wchevreuil pushed a commit
that referenced
this pull request
Sep 1, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
wchevreuil pushed a commit
that referenced
this pull request
Sep 3, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: If213337be959a392a9bc55aba63b4d033df8e729
wchevreuil pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Sep 3, 2025
…t-cache (apache#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: If213337be959a392a9bc55aba63b4d033df8e729
wchevreuil pushed a commit
that referenced
this pull request
Sep 3, 2025
…t-cache (#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: If213337be959a392a9bc55aba63b4d033df8e729
stoty pushed a commit
to stoty/hbase
that referenced
this pull request
Nov 22, 2025
…into cdh_main HBASE-28465 Implementation of framework for time-based priority bucket-cache (apache#5793) Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org> Change-Id: If213337be959a392a9bc55aba63b4d033df8e729
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.