HBASE-27570 Unify tracking of block IO across all read request types by bbeaudreault · Pull Request #5004 · apache/hbase

bbeaudreault · 2023-02-01T20:32:22Z

There are two benefits to this change:

The old model totally ignores blocks retained by filtered cells. Filtered blocks are still retained, so should be counted.
The old model required covering all request types with addSize calls, and missed checkAndMutate/append/increment. The new model simplifies and unifies accounting between Multiget/Scan and makes it possible to consistently report accurate block IO in metrics (future PR).

However, there is one important functional change here. Previously we kept track of lastBlock in order to try not to overcount blocks when submitting multigets for individual cells of the same row or multiple contiguous rows in the same block. For example:

Block A contains rows 1, 2, 3
Submit table.get(List.of(1, 2, 3))
Existing code would only count block A's size once, while new code will count it 3 times.

I could add similar handling by stashing lastBlock in RpcCallContext (see example commit here). For measuring work done by a request, the new solution is more accurate as-is. For limiting retained blocks, the overcounting may just cause some requests to be broken into smaller chunks than they are today.

The default block size is 64kb and default max result size is 100mb. So one would need to submit a multiget with 1600 gets for individual columns of a row or perfectly ordered contiguous rows within 1 block in order to run into a difference. This feels like an edge case, and lastBlock tracking has numerous ways it can mess up:

If the table in the above example had 2 CFs, the same example would overcount in existing solution because each row would have 2 blocks. So lastBlock would keep swapping between those 2 blocks and not actually work.
If rows get writes over time, a row is likely to be spread across multiple StoreFiles/blocks and the same above would happen even with 1 CF.
If the multiget batch has rows in multiple blocks, and the batch is not ordered by row order. For example block A contains rows 1 and 2, block B contains row 3. Submitting a batch for 1, 3, 2 and lastBlock would be A - B - A, so A would get counted twice
If setCacheBlocks(false), the equality check would not work.

With those caveats, it did not feel worth adding the complexity of RpcCallContext lastBlock tracking. But I could easily be convinced otherwise.

bbeaudreault · 2023-02-03T01:15:19Z

+ // This get returns nothing since the filter doesn't match. Filtered cells still retain
+ // blocks, and this is a full row scan of both blocks. This equals 100 bytes so we should
+ // throw a multiResponseTooLarge after this get if we are counting filtered cells correctly.
+ Get g0 = new Get(row).addFamily(FAMILY).setFilter(
+ new QualifierFilter(CompareOperator.EQUAL, new BinaryComparator(Bytes.toBytes("sdf"))));


old method removed from addSize would not count any block io for this get.

bbeaudreault · 2023-02-03T01:16:30Z

+ // g1 and g2 each count the first 55 byte block, so we end up with block size of 110
+ // after g2 and throw a multiResponseTooLarge before g3
+ Get g1 = new Get(row);
+ g1.addColumn(FAMILY, cols[0]);
+ gets.add(g1);
+
 Get g2 = new Get(row);
 g2.addColumn(FAMILY, cols[3]);
 gets.add(g2);


old method would have returned these 2 gets in the same rpc, because of lastBlock tracking (both columns in the same block). We could add that functionality back with a change like this

but this only works because of the order of the gets g1, g2, g3. if we instead ordered it g1, g3, g2 lastBlock would not help.

Apache-HBase · 2023-02-03T01:36:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	patch	0m 3s	#5004 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem	Report/Notes
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-02-03T01:36:16Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	docker	0m 0s	Docker failed to build yetus/hbase:e7d1a6a052.

Subsystem	Report/Notes
GITHUB PR	#5004
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/5/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault · 2023-02-03T01:44:11Z

Build broken due to https://issues.apache.org/jira/browse/HBASE-27608. Will rebase once that's merged.

Apache-HBase · 2023-02-03T05:20:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 21s	Docker mode activated.
-0 ⚠️	yetus	0m 4s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 56s	master passed
+1 💚	compile	0m 39s	master passed
+1 💚	shadedjars	4m 16s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 42s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	4m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	203m 46s	hbase-server in the patch passed.
		224m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5004
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 907543e9bb83 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `55b4bbc`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/5/testReport/
Max. process+thread count	2462 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/5/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-02-03T18:05:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 59s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 39s	master passed
+1 💚	compile	2m 24s	master passed
+1 💚	checkstyle	0m 34s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 27s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 11s	the patch passed
+1 💚	compile	2m 19s	the patch passed
+1 💚	javac	2m 19s	the patch passed
+1 💚	checkstyle	0m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 20s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 37s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 13s	The patch does not generate ASF License warnings.
		37m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5004
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux ec972a4be251 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `bbc20f3`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-02-03T21:12:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	0m 38s	master passed
+1 💚	shadedjars	4m 15s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 43s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 15s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	203m 41s	hbase-server in the patch passed.
		224m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5004
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 36deaa319c14 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `bbc20f3`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/testReport/
Max. process+thread count	2343 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-02-03T21:15:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 18s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 41s	master passed
+1 💚	compile	0m 46s	master passed
+1 💚	shadedjars	4m 25s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 17s	the patch passed
+1 💚	compile	0m 47s	the patch passed
+1 💚	javac	0m 47s	the patch passed
+1 💚	shadedjars	4m 25s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	204m 24s	hbase-server in the patch passed.
		227m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5004
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4bfeab1f54e8 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `bbc20f3`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/testReport/
Max. process+thread count	2444 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5004/6/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9

Nice clean up!

…5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

…pache#5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

…ead request types (apache#5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

This comment was marked as outdated.
Sign in to view

bbeaudreault force-pushed the HBASE-27570-2 branch from 2ee8826 to 786f2b4Compare February 2, 2023 16:29

This comment was marked as outdated.
Sign in to view

bbeaudreault commented Feb 3, 2023
View reviewed changes

This comment was marked as outdated.
Sign in to view

bbeaudreault force-pushed the HBASE-27570-2 branch from 91e02cf to 7e0ef57Compare February 3, 2023 01:25

bbeaudreault marked this pull request as ready for review February 3, 2023 01:29

bbeaudreault added 2 commits February 3, 2023 11:57

HBASE-27570 Unify tracking of block IO across all read request types
9075633

extend test to include filtered case and show lastBlock implications
a4da6f1

bbeaudreault force-pushed the HBASE-27570-2 branch from 7e0ef57 to a4da6f1Compare February 3, 2023 16:57

Apache9 approved these changes Feb 4, 2023
View reviewed changes

bbeaudreault merged commit 7b26b2a into apache:masterFeb 4, 2023

bbeaudreault deleted the HBASE-27570-2 branch February 4, 2023 17:14

bbeaudreault added a commit that referenced this pull request Feb 4, 2023
HBASE-27570Unify tracking of block IO across all read request types (#…
8809c88
…5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Feb 5, 2023
HBASE-27570 Unify tracking of block IO across all read request types (a…
9263971
…pache#5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Feb 10, 2023
HBASE-27570 Unify tracking of block IO across all read request types (a…
1bc4297
…pache#5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Feb 10, 2023
HBASE-27570 Unify tracking of block IO across all read request types (a…
84d5a65
…pache#5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Feb 28, 2023
HubSpot Backport: HBASE-27570 Unify tracking of block IO across all r…
047cd16
…ead request types (apache#5004) Signed-off-by: Duo Zhang <zhangduo@apache.org>

Uh oh!

Conversation

bbeaudreault commented Feb 1, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

bbeaudreaultFeb 3, 2023

Choose a reason for hiding this comment

Uh oh!

bbeaudreaultFeb 3, 2023

Choose a reason for hiding this comment

Uh oh!

bbeaudreaultFeb 3, 2023

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Apache-HBase commented Feb 3, 2023

Uh oh!

Apache-HBase commented Feb 3, 2023

Uh oh!

bbeaudreault commented Feb 3, 2023

Uh oh!

Apache-HBase commented Feb 3, 2023

Uh oh!

Apache-HBase commented Feb 3, 2023

Uh oh!

Apache-HBase commented Feb 3, 2023

Uh oh!

Apache-HBase commented Feb 3, 2023

Uh oh!

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bbeaudreault commented Feb 1, 2023•
edited
Loading