Uh oh!
There was an error while loading. Please reload this page.
HBASE-27686: Recovery of BucketCache and Prefetched data after RS Crash - #5080
Merged
Conversation
Kota-SH
marked this pull request as draft
March 3, 2023 17:55
Apache-HBase
commented
Mar 3, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 3, 2023
💔 -1 overall
This message was automatically generated. |
wchevreuil
left a comment
Contributor
There was a problem hiding this comment.
I think this is a great, yet simple solution for the persistent cache map. We could also solve the problem of evicted blocks by setting cacheDirty to true after we remove cache keys from the map here.
Comment on lines
+475
to
+477
| if (ioEngine.isPersistent()){ | ||
| setCacheDirty(true); | ||
| } |
Contributor
There was a problem hiding this comment.
At this point we are not guaranteed to have added the block, as it's just sent to the writer threads. We should be doing this here instead.
Uh oh!
There was an error while loading. Please reload this page.
| private final BucketCacheStats cacheStats = new BucketCacheStats(); | ||
| private final String persistencePath; | ||
| boolean isCacheDirty; |
| this.backingMap = new ConcurrentHashMap<>((int) blockNumCapacity); | ||
| if (ioEngine.isPersistent() && persistencePath != null) { | ||
| BucketCachePersister cachePersister = new BucketCachePersister(this, 1000); |
Contributor
There was a problem hiding this comment.
Make this interval configurable. Maybe worth decreasing this default.
| Thread.sleep(1000); | ||
| } | ||
| // Kill the RS | ||
| cluster.killRegionServer(cluster.getRegionServer(0).getServerName()); |
Contributor
There was a problem hiding this comment.
I had noticed that even when using this killRegionServer, we still go through the normal shutdown path for block cache, so we can't really rely on this. But since the main logic we want to test is that updates happen while the RS is running, we could check for the file existence without killing the RS. Also, can we validate the contents of the persisted file? At this point, I think we should certify that all the cache entries are there.
Apache-HBase
commented
Mar 14, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 14, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 14, 2023
💔 -1 overall
This message was automatically generated. |
Kota-SH
marked this pull request as ready for review
March 14, 2023 17:11
Apache-HBase
commented
Mar 14, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 14, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 14, 2023
💔 -1 overall
This message was automatically generated. |
wchevreuil
left a comment
Contributor
There was a problem hiding this comment.
I think we are almost there, just some minor comments on the tests.
| prefetchPersistencePath = testDir + "/prefetch.persistence"; | ||
| conf.set(CacheConfig.PREFETCH_PERSISTENCE_PATH_KEY, prefetchPersistencePath); | ||
| FileSystem fs = HFileSystem.get(conf); | ||
| long BUCKETCACHE_PERSIST_INTERVAL = 3000; |
Contributor
There was a problem hiding this comment.
Can we decrease this interval to a sub-second value and make this test more expedite?
ContributorAuthor
There was a problem hiding this comment.
So writeStoreFile() and readStoreFile() methods take at least 2400 millis. So I used a value higher than that.
| prefetchPersistencePath = testDir + "/prefetch.persistence"; | ||
| conf.set(CacheConfig.PREFETCH_PERSISTENCE_PATH_KEY, prefetchPersistencePath); | ||
| FileSystem fs = HFileSystem.get(conf); | ||
| long BUCKETCACHE_PERSIST_INTERVAL = 3000; |
Contributor
There was a problem hiding this comment.
Let's use camel case for local variable naming.
| Path storeFile2 = writeStoreFile("TestPrefetch1", conf, cacheConf, fs); | ||
| readStoreFile(storeFile, 0, fs, cacheConf, conf, bucketCache); | ||
| readStoreFile(storeFile2, 0, fs, cacheConf, conf, bucketCache); | ||
| Thread.sleep(BUCKETCACHE_PERSIST_INTERVAL - startTime + System.currentTimeMillis() + 100); |
Contributor
There was a problem hiding this comment.
So readStoreFile makes sure to only return once prefetch is complete. Do we still this extra (- startTime + System.currentTimeMillis()), plus the 100 milis to pass this test?
| final int writerQLen = BucketCache.DEFAULT_WRITER_QUEUE_ITEMS; | ||
| @Test | ||
| public void testPrefetchPersistenceCrash() throws Exception { |
Contributor
There was a problem hiding this comment.
The two test methods are almost identical. Can we do some code reuse with some lambdas for the different pre-conditions/predicates?
wchevreuil
commented
Mar 15, 2023
Contributor
Lets also update the flag for evictions. That way, we would sort the need for update on evictions as well. |
Kota-SHforce-pushed
the
hbase-27686
branch
2 times, most recently
from
March 15, 2023 17:47
e252244 to
52b230bCompareApache-HBase
commented
Mar 15, 2023
💔 -1 overall
This message was automatically generated. |
| cache.setCacheInconsistent(false); | ||
| } | ||
| } catch (IOException | InterruptedException e) { | ||
| LOG.info("Exception in BucketCachePersister" + e.getMessage()); |
| private static final Logger LOG = LoggerFactory.getLogger(BucketCachePersister.class); | ||
| public BucketCachePersister(BucketCache cache, long intervalMillis) { | ||
| this.cache = cache; |
Contributor
There was a problem hiding this comment.
Nit: Let's set a meaningful thread name for this one.
Apache-HBase
commented
Mar 15, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 15, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 16, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 16, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 16, 2023
🎊 +1 overall
This message was automatically generated. |
wchevreuil
approved these changes
Mar 16, 2023
Apache-HBase
commented
Mar 16, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 16, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 16, 2023
💔 -1 overall
This message was automatically generated. |
vinayakphegde pushed a commit
to vinayakphegde/hbase
that referenced
this pull request
Apr 4, 2024
…tched data after RS Crash (apache#5080) Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org> (cherry picked from commit 58cb1f4) Change-Id: I0d446b51f16f11d91ab8a6619ce5b01f8de7da91
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-27686