Uh oh!
There was an error while loading. Please reload this page.
HBASE-25392 Direct insert compacted HFiles into data directory. - #3389
Merged
Conversation
wchevreuil
requested review from
Apache9, anoopsjohn, apurtell, joshelser, petersomogyi, saintstack and taklwuJune 15, 2021 12:52
Apache-HBase
commented
Jun 15, 2021
💔 -1 overall
This message was automatically generated. |
Change-Id: Iff62fd5d402429115b743dd22293c357cbf1a29e
Apache-HBase
commented
Jun 15, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 15, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 16, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 16, 2021
🎊 +1 overall
This message was automatically generated. |
petersomogyi
left a comment
Contributor
There was a problem hiding this comment.
Nice change. I posted some minor comments.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Change-Id: I5ac86a27c87e956926860ad9fb3eaeed3be75d00
Apache-HBase
commented
Jun 21, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 21, 2021
💔 -1 overall
This message was automatically generated. |
z-york
requested changes
Jun 22, 2021
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Comment on lines
+59
to
+61
| if (store.getHRegion().getRegionServerServices() != null) { | ||
| favoredNodes = store.getHRegion().getRegionServerServices().getFavoredNodesForRegion( | ||
| store.getHRegion().getRegionInfo().getEncodedName()); |
Contributor
There was a problem hiding this comment.
Why not use this directly from the StoreContext: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreContext.java#L95
ContributorAuthor
There was a problem hiding this comment.
I already need an instance of HStore, as you noted on your comment below. This HStore instance need is not exclusive from this Compactor extension, it is already defined on the parent classes single declared constructors, so it's used for other stuff in the hierarchy, not only the filereader creation. I don't think there's much value in create an extra StoreContext here when we have everything we need in HStore instance itself.
Contributor
There was a problem hiding this comment.
The StoreContext is initialized with the store so there is no need to create a context. I'm specifically suggesting to use the store.getStoreContext().getFavoredNodes() instead of having to expose the RSServices and have all this extra code here.
ContributorAuthor
There was a problem hiding this comment.
Wasn't finding store.getStoreContext() before, since it was package private. Made it public on HStore and am calling it there now.
Uh oh!
There was an error while loading. Please reload this page.
Change-Id: I7a79666b8856715b1ad4421162d2b4658c213f4d
Apache-HBase
commented
Jun 22, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 22, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 22, 2021
🎊 +1 overall
This message was automatically generated. |
z-york
requested changes
Jun 22, 2021
| * @param totalCompactedFilesSize the total size of compacted files. | ||
| * @return true if the checks mentioned above pass and the cache is enabled, false otherwise. | ||
| */ | ||
| public boolean enableCacheOnWrite(long totalCompactedFilesSize) { |
Contributor
There was a problem hiding this comment.
Should we include something to distinguish that this is specifically for compactions in the method name since the CacheConfig is used in more places than just compactions?
ContributorAuthor
There was a problem hiding this comment.
Renamed it on last commit.
| Compression.Algorithm compression, boolean includeMVCCReadpoint, boolean includesTag, | ||
| boolean shouldDropBehind, long totalCompactedFilesSize) throws IOException { | ||
| final CacheConfig writerCacheConf; | ||
| // Don't cache data on write on compactions. |
Contributor
There was a problem hiding this comment.
Update/remove this comment since now it depends on the configs
Change-Id: I26d10e3cb52a14bc82f13647ff9921c92282a353
Apache-HBase
commented
Jun 23, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jun 24, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 5, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 5, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 5, 2021
💔 -1 overall
This message was automatically generated. |
Change-Id: I40ca4a799f9f4a4e406f34a637cc75ec765a21d9
Apache-HBase
commented
Jul 6, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 6, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 6, 2021
🎊 +1 overall
This message was automatically generated. |
joshelser
requested changes
Jul 19, 2021
joshelser
left a comment
Member
There was a problem hiding this comment.
2nd pass (I was running out the door the first time I looked this over and was going to be repeating a bit from Stack, so I abstained).
It's nice to see this coming together, Wellington. Let's see if we can think through how we can clean up some of the smells Stack points out.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| } | ||
| protected HStoreFile createStoreFileAndReader(final Path p) throws IOException { | ||
| public HStoreFile createStoreFileAndReader(final Path p) throws IOException { |
Member
There was a problem hiding this comment.
I keep trying to think about what the right abstraction to have for our internal API.
Is there a reason to not have the pluggable compaction code have a method like this to return the HStoreFile? Glancing, I don't see any obvious methods/classes which are package-private to HStore (that would require these methods to live here).
Although, I also see that HStore#openStoreFiles is using one instance of createStoreFileAndReader, so we couldn't completely move this out of HStore. Is it better to lift one method into Store and leave other instances "internal" to HStore? That might help the smell a little, but an obvious solution isn't jumping out at me.
| } | ||
| HFileContext createFileContext(Compression.Algorithm compression, | ||
| public HFileContext createFileContext(Compression.Algorithm compression, |
Member
There was a problem hiding this comment.
re-implement this in the DirectStoreCompactor
I feel like that would just result in having to change code in multiple places which is error-prone. What about moving this onto HFileContext itself, or the Builder. Create a method which lets you pass the Builder an Store and do most of this work there. That seems to be more appropriate to me (at a glance) than doing this in HStore.
| public long minSeqIdToKeep = 0; | ||
| /** Total size of the compacted files **/ | ||
| private long totalCompactedFilesSize = 0; | ||
| public long totalCompactedFilesSize = 0; |
Member
There was a problem hiding this comment.
suggestion: leave the variable private, make a public (if necessary) getter for this method, and make a package-private/private method to mutate the value.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| when(mockStore.getRegionFileSystem()).thenReturn(mockRegionFS); | ||
| when(mockRegionFS.getRegionDir()).thenReturn(filePath); | ||
| when(mockStore.getColumnFamilyName()).thenReturn(cfName); | ||
| HFileContext mockFileContext = mock(HFileContext.class); |
Member
There was a problem hiding this comment.
What about making a real HFileContext to avoid ~5 mockings?
| when(mockStore.getHRegion()).thenReturn(mock(HRegion.class)); | ||
| when(mockStore.getStoreContext()).thenReturn(new StoreContext.Builder(). | ||
| withFavoredNodesSupplier(()-> null).build()); | ||
| ColumnFamilyDescriptor mockDesc = mock(ColumnFamilyDescriptor.class); |
Uh oh!
There was an error while loading. Please reload this page.
Change-Id: I618ba247caf013c30eb54ea9bf0e36d0d1baabe3
Apache-HBase
commented
Jul 19, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 19, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 19, 2021
💔 -1 overall
This message was automatically generated. |
Change-Id: I120093272991de10d33f02903e7b5e51c6767978
Apache-HBase
commented
Jul 20, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 20, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 20, 2021
🎊 +1 overall
This message was automatically generated. |
Change-Id: Ie439d3a0dc6d6e5006c37c72711b0f236aa38908
Apache-HBase
commented
Jul 21, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 21, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 21, 2021
🎊 +1 overall
This message was automatically generated. |
joshelser
requested changes
Jul 21, 2021
joshelser
left a comment
Member
There was a problem hiding this comment.
I think migrating the Function to some kind of named and documented interface is the only thing I see remaining. Good work, Wellington. I think I'm +1 after this.
My comment about StripeCompactor and DTCompactor can be deferred, I think. Doesn't need to complicate this further.
| p -> { | ||
| try { | ||
| return this.createStoreFileAndReader((Path) p); | ||
| }catch(IOException e){ |
Member
There was a problem hiding this comment.
nit whitespace/checkstyle
but I like the solution!
Uh oh!
There was an error while loading. Please reload this page.
…ctor Change-Id: I129da2790d72e0a1043f6359704bc9fc423bcc30
wchevreuil
commented
Jul 22, 2021
ContributorAuthor
Yeah, had thought about it previously. Had just added it in the last commit.
Agreed. Right now, SC and DTC will only work with the default engine that does temp/rename. We can work later on support direct store for these type of compactions too. |
Apache-HBase
commented
Jul 22, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 22, 2021
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 22, 2021
🎊 +1 overall
This message was automatically generated. |
joshelser
approved these changes
Jul 23, 2021
joshelser
left a comment
Member
There was a problem hiding this comment.
+1 on the latest changes here. Looks like we have some flakey tests, but only one of them didn't pass on re-run
[ERROR] org.apache.hadoop.hbase.master.TestClusterRestartFailover.test
[ERROR] Run 1: TestClusterRestartFailover.test:130 serverNode should be deleted after SCP finished expected null, but was:<f286fa1f49d4,39095,1626965264700/OFFLINE/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@4a62266f[Write locks = 0, Read locks = 0]>
[ERROR] Run 2: TestClusterRestartFailover.test:134 Metrics Counters should be equal expected:<4> but was:<8>
[ERROR] Run 3: TestClusterRestartFailover.test:134 Metrics Counters should be equal expected:<4> but was:<12>
@saintstack and @z-york, what do you think about this change at this point? Want Wellington to wait on your re-reviews?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continuing with the works of HBASE-24749. This one is about compactions, with the introduction of DirectInStoreCompactor that allows for writing compacted files direct in the store dir.
I had also renamed the Flusher and FlushContext classes introduced by HBASE-25391 to have more meaningful naming.