Skip to content

HBASE-27795: Define RPC API for cache cleaning - #5525

Closed
Kota-SH wants to merge 1 commit into
apache:HBASE-27389from
Kota-SH:hbase-27398-new
Closed

HBASE-27795: Define RPC API for cache cleaning#5525
Kota-SH wants to merge 1 commit into
apache:HBASE-27389from
Kota-SH:hbase-27398-new

Conversation

@Kota-SH

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 36sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗prototool0m 0sprototool was not available.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ HBASE-27389 Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall2m 48sHBASE-27389 passed
+1 💚compile4m 25sHBASE-27389 passed
+1 💚checkstyle1m 30sHBASE-27389 passed
+1 💚spotless0m 44sbranch has no errors when running spotless:check.
+1 💚spotbugs5m 34sHBASE-27389 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for patch
+1 💚mvninstall2m 35sthe patch passed
+1 💚compile4m 26sthe patch passed
+1 💚cc4m 26sthe patch passed
-0 ⚠️javac2m 28shbase-server generated 1 new + 194 unchanged - 1 fixed = 195 total (was 195)
-0 ⚠️checkstyle0m 37shbase-server: The patch generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 17sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚hbaseprotoc1m 40sthe patch passed
-1 ❌spotless0m 38spatch has 21 errors when running spotless:check, run spotless:apply to fix.
-1 ❌spotbugs1m 41shbase-server generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
_ Other Tests _
+1 💚asflicense0m 40sThe patch does not generate ASF License warnings.
49m 16s
ReasonTests
FindBugsmodule:hbase-server
org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.uncacheStaleBlocks(HRegionServer) makes inefficient use of keySet iterator instead of entrySet iterator At BucketCache.java:keySet iterator instead of entrySet iterator At BucketCache.java:[line 2015]
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5525
JIRA IssueHBASE-27795
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool
unameLinux 23233cf2b95d 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-27389 / 69d980a
Default JavaEclipse Adoptium-11.0.17+8
javachttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt
checkstylehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotlesshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-general-check/output/patch-spotless.txt
spotbugshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-general-check/output/new-spotbugs-hbase-server.html
Max. process+thread count81 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-27389 Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for branch
+1 💚mvninstall2m 23sHBASE-27389 passed
+1 💚compile1m 31sHBASE-27389 passed
+1 💚shadedjars4m 48sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 1sHBASE-27389 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for patch
+1 💚mvninstall2m 12sthe patch passed
+1 💚compile1m 31sthe patch passed
+1 💚javac1m 31sthe patch passed
+1 💚shadedjars4m 47spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 59sthe patch passed
_ Other Tests _
+1 💚unit0m 27shbase-protocol-shaded in the patch passed.
+1 💚unit1m 19shbase-client in the patch passed.
-1 ❌unit236m 33shbase-server in the patch failed.
+1 💚unit7m 19shbase-thrift in the patch passed.
270m 16s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5525
JIRA IssueHBASE-27795
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9f52196052c5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-27389 / 69d980a
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/testReport/
Max. process+thread count4636 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 15sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-27389 Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall2m 48sHBASE-27389 passed
+1 💚compile2m 10sHBASE-27389 passed
+1 💚shadedjars4m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 24sHBASE-27389 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall2m 42sthe patch passed
+1 💚compile2m 9sthe patch passed
+1 💚javac2m 9sthe patch passed
+1 💚shadedjars4m 41spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 24sthe patch passed
_ Other Tests _
+1 💚unit0m 39shbase-protocol-shaded in the patch passed.
+1 💚unit1m 41shbase-client in the patch passed.
-1 ❌unit243m 35shbase-server in the patch failed.
+1 💚unit6m 42shbase-thrift in the patch passed.
280m 12s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5525
JIRA IssueHBASE-27795
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2d94bd77166c 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-27389 / 69d980a
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/testReport/
Max. process+thread count5167 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗prototool0m 0sprototool was not available.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ HBASE-27389 Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for branch
+1 💚mvninstall2m 51sHBASE-27389 passed
+1 💚compile4m 32sHBASE-27389 passed
+1 💚checkstyle1m 25sHBASE-27389 passed
+1 💚spotless0m 45sbranch has no errors when running spotless:check.
+1 💚spotbugs5m 35sHBASE-27389 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for patch
+1 💚mvninstall2m 40sthe patch passed
+1 💚compile4m 23sthe patch passed
+1 💚cc4m 23sthe patch passed
+1 💚javac4m 23sthe patch passed
-0 ⚠️checkstyle0m 38shbase-server: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 17sPatch does not cause any errors with Hadoop 3.2.4 3.3.5.
+1 💚hbaseprotoc1m 40sthe patch passed
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs6m 6sthe patch passed
_ Other Tests _
+1 💚asflicense0m 38sThe patch does not generate ASF License warnings.
49m 41s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5525
JIRA IssueHBASE-27795
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile cc hbaseprotoc prototool
unameLinux fc90fc1b2775 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-27389 / 69d980a
Default JavaEclipse Adoptium-11.0.17+8
checkstylehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count81 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 33sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-27389 Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for branch
+1 💚mvninstall2m 32sHBASE-27389 passed
+1 💚compile1m 36sHBASE-27389 passed
+1 💚shadedjars4m 59sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 3sHBASE-27389 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for patch
+1 💚mvninstall2m 12sthe patch passed
+1 💚compile1m 36sthe patch passed
+1 💚javac1m 36sthe patch passed
+1 💚shadedjars4m 55spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 1sthe patch passed
_ Other Tests _
+1 💚unit0m 27shbase-protocol-shaded in the patch passed.
+1 💚unit1m 20shbase-client in the patch passed.
-1 ❌unit231m 43shbase-server in the patch failed.
+1 💚unit6m 36shbase-thrift in the patch passed.
265m 22s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5525
JIRA IssueHBASE-27795
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 8d1a61d4f139 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-27389 / 69d980a
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/testReport/
Max. process+thread count4663 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 17sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-27389 Compile Tests _
+0 🆗mvndep0m 11sMaven dependency ordering for branch
+1 💚mvninstall2m 48sHBASE-27389 passed
+1 💚compile2m 10sHBASE-27389 passed
+1 💚shadedjars4m 41sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 25sHBASE-27389 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall2m 38sthe patch passed
+1 💚compile2m 10sthe patch passed
+1 💚javac2m 10sthe patch passed
+1 💚shadedjars4m 38spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 22sthe patch passed
_ Other Tests _
+1 💚unit0m 38shbase-protocol-shaded in the patch passed.
+1 💚unit1m 42shbase-client in the patch passed.
-1 ❌unit229m 38shbase-server in the patch failed.
+1 💚unit6m 55shbase-thrift in the patch passed.
266m 30s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5525
JIRA IssueHBASE-27795
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 50a87d11e864 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-27389 / 69d980a
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/testReport/
Max. process+thread count4712 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5525/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will happen if the region is moved back while we are cleaning its blocks in block cache?
And we only implement the uncache method for BucketCache? What about other types of BlockCache?

* @param server HRegionServer
* @return A map of filename and number of blocks evicted.
*/
default Optional<Map<String, Integer>> uncacheStaleBlocks(HRegionServer server) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer here we pass an interface, for testing whether a region is available, instead of pass a HRegionServer directly. It will be easier for testing.
And just returning a Map is enough? If we do not clean any blocks, just return an empty map?

} else {
blocksEvicted = 0;
}
} catch (NotServingRegionException nsre) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the normal path? In the above if condition, we test isAvailable, so why here we could still get a NotServingRegionException?

@Kota-SH

Copy link
Copy Markdown
ContributorAuthor

@Apache9 - Thanks for the review.

What will happen if the region is moved back while we are cleaning its blocks in block cache?
As of now, we check the region state before calling evictBlocksByHfileName() method in uncacheStaleBlocks(). So we only evict blocks if the region state isClosed/isClosing. If the region is moved back, it will be in a different state, and we do not evict its blocks.

What about other types of BlockCache?
Currently, we only support BucketCache and CombinedBlockCache. We can extend this feature to other types of block cache if required. I can create a separate jira to handle those scenarios.

I prefer here we pass an interface / If we do not clean any blocks, just return an empty map?
Yes, we pass an empty map if we don't evict any blocks. I made the change to pass an Interface in a separate PR to the master branch (#5492). Could you please review it there?

Thanks,
Shanmukha K

@Kota-SHKota-SH closed this Dec 4, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@Kota-SH@Apache-HBase@Apache9@shankot