HBASE-27241 Add metrics for evaluating cost and effectiveness of bloom filters by bbeaudreault · Pull Request #4669 · apache/hbase

bbeaudreault · 2022-07-30T13:56:26Z

Add static index/bloom sizes
Add metrics for bloom requests and negative results

Combined these together are very useful for determining whether you should add bloom filters (if eligible requests is high) and whether they are worth the cost (compare bloom size to bloom request/negative result count)

I added a test in TestHStore because I mostly wanted to verify that these new request/negative/eligible counts make it through the entire stack end-to-end, since it involves plumbing into a few classes.

…m filters - Add static index/bloom sizes - Add metrics for bloom requests and negative results

Apache-HBase · 2022-07-30T14:40:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 55s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 39s	master passed
+1 💚	compile	3m 19s	master passed
+1 💚	checkstyle	1m 3s	master passed
+1 💚	spotless	0m 58s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 0s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 32s	the patch passed
+1 💚	compile	2m 57s	the patch passed
-0 ⚠️	javac	0m 20s	hbase-hadoop-compat generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
-0 ⚠️	checkstyle	0m 8s	hbase-hadoop-compat: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-0 ⚠️	checkstyle	0m 34s	hbase-server: The patch generated 1 new + 9 unchanged - 1 fixed = 10 total (was 10)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	13m 18s	Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚	spotless	0m 48s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 48s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 15s	The patch does not generate ASF License warnings.
		39m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4669
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux ebe39f23bfad 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e8c14ee`
Default Java	AdoptOpenJDK-1.8.0_282-b08
javac	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-general-check/output/diff-compile-javac-hbase-hadoop-compat.txt
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-hadoop-compat.txt
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-hadoop-compat hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-30T17:43:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 22s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 22s	master passed
+1 💚	compile	0m 53s	master passed
+1 💚	shadedjars	3m 43s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 16s	the patch passed
+1 💚	compile	0m 52s	the patch passed
+1 💚	javac	0m 52s	the patch passed
+1 💚	shadedjars	3m 42s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 34s	hbase-hadoop-compat in the patch passed.
+1 💚	unit	205m 32s	hbase-server in the patch passed.
		223m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4669
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1fcdcbf118ad 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e8c14ee`
Default Java	AdoptOpenJDK-1.8.0_282-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/testReport/
Max. process+thread count	2601 (vs. ulimit of 30000)
modules	C: hbase-hadoop-compat hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-07-30T17:49:20Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 5s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 51s	master passed
+1 💚	compile	1m 1s	master passed
+1 💚	shadedjars	3m 47s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 37s	the patch passed
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
+1 💚	shadedjars	3m 45s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 41s	the patch passed
		_ Other Tests _
+1 💚	unit	0m 38s	hbase-hadoop-compat in the patch passed.
-1 ❌	unit	208m 46s	hbase-server in the patch failed.
		229m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4669
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 0ffefba27edd 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `e8c14ee`
Default Java	AdoptOpenJDK-11.0.10+9
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/testReport/
Max. process+thread count	2387 (vs. ulimit of 30000)
modules	C: hbase-hadoop-compat hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4669/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk

Nothing terribly surprising here, and I learned a bit about how metric aggregation at the table level works. Thanks!

bbeaudreault · 2022-08-09T19:19:06Z

Thanks Nick!

…m filters (#4669) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

HBASE-27241 Add metrics for evaluating cost and effectiveness of bloo…
468cdc5
…m filters - Add static index/bloom sizes - Add metrics for bloom requests and negative results

ndimiduk approved these changes Aug 9, 2022
View reviewed changes

bbeaudreault merged commit bfa53f5 into apache:masterAug 9, 2022

bbeaudreault deleted the HBASE-27241 branch August 9, 2022 19:18

bbeaudreault added a commit that referenced this pull request Aug 9, 2022
HBASE-27241Add metrics for evaluating cost and effectiveness of bloo…
e99faad
…m filters (#4669) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

bbeaudreault added a commit that referenced this pull request Aug 9, 2022
HBASE-27241Add metrics for evaluating cost and effectiveness of bloo…
c0717c7
…m filters (#4669) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>

Navigation Menu

Uh oh!

HBASE-27241 Add metrics for evaluating cost and effectiveness of bloom filters - #4669

HBASE-27241 Add metrics for evaluating cost and effectiveness of bloom filters#4669
bbeaudreault merged 1 commit into
apache:masterfrom
HubSpot:HBASE-27241

bbeaudreault commented Jul 30, 2022•
edited
Loading

Uh oh!

Apache-HBase commented Jul 30, 2022

Uh oh!

Apache-HBase commented Jul 30, 2022

Uh oh!

Apache-HBase commented Jul 30, 2022

Uh oh!

ndimiduk left a comment

Uh oh!

bbeaudreault commented Aug 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bbeaudreault commented Jul 30, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Jul 30, 2022

Uh oh!

Apache-HBase commented Jul 30, 2022

Uh oh!

Apache-HBase commented Jul 30, 2022

Uh oh!

ndimiduk left a comment

Choose a reason for hiding this comment

Uh oh!

bbeaudreault commented Aug 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bbeaudreault commented Jul 30, 2022•
edited
Loading