HBASE-28187 NPE when flushing a non-existing column family by guluo2016 · Pull Request #5692 · apache/hbase

guluo2016 · 2024-02-19T11:55:53Z

And even worse, although this flush can be terminated by exception or manually，we are still unable to operate this region because of RegionTooBusyException.

ndimiduk · 2024-09-18T14:30:08Z

+ "There are non-existing families %s, we cannot flush the region %s, in table %s.",
+ noSuchFamilies, getRegionInfo().getRegionNameAsString(),
+ getTableDescriptor().getTableName().getNameAsString());
+ LOG.warn(noSuchFamiliesMsg);


I don't think this should log at all. It's essentially an HTTP 400 kind of situation. Maybe a TRACE log, if at all. By all means, send the message back to the client, but i don't think the RS needs to emit this message.

I also think the log here is unnecessary too. If you want to log this to trace bad requests, RegionServer will log some key information for FlushRegionRequest and ExecuteProceduresRequest. Is it possible to merge the information here into that log?

ndimiduk · 2024-09-18T14:32:18Z

- }
- });
+ TableName tableName = location.getRegion().getTable();
+ addListener(getDescriptor(tableName), (tDesc, error2) -> {


See above about introducing the additional RPC.
IMHO, the client should be dumber, not smarter. I'd rather see the NoSuchColumnFamilyException come from the RS than the client.

Agree. I think an NoSuchColumnFamilyException here would be better too.

ndimiduk · 2024-09-18T14:43:01Z

- (resp) -> resp.getProcId(), new FlushTableProcedureBiConsumer(tableName));
 CompletableFuture<Void> future = new CompletableFuture<>();
- addListener(procFuture, (ret, error) -> {
+ addListener(getDescriptor(tableName), (tDesc, error) -> {


Introducing getDescriptors introduces an additional RPC. I think we shouldn't do this, only to do validation on client-side. Instead please post the original message and let the server validate it. If we had some client-side cache of the descriptor, we could do a pre-validation on that, but even then, caches are lossy and probably shouldn't be relied upon for something that the server is already the authority.

I will refactor this part of the code later based on your good suggestions
Thank you both for your review

guluo2016 · 2024-09-20T12:27:18Z

I hava already refactor these code, please review, thanks. @ndimiduk @frostruan @Apache9

Summary:

flush 'table', 'non_existing_family'

Flush Procedure is enabled.

we would check if the column families exist when executing FlushTableProcedure, and throwing NoSuchColumnFamilyException if it doesn't exist. And we will not retry to flush again at client side.

Flush Procedure is disabled.

we would check if the column families exist on MasterFlushTableProcedureManager.execProcedure(), and throwing NoSuchColumnFamilyException if it doesn't exist.

flush 'region_name', 'non_existing_family'

we would check if the column families exist on RSRpcServices.flushRegion(), and throwing NoSuchColumnFamilyException if it doesn't exist.

frostruan · 2024-09-21T02:08:18Z

LGTM. +1

ndimiduk

This looks better. Thanks @guluo2016 .

Apache-HBase · 2024-09-23T16:40:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 15s		master passed
+1 💚	compile	3m 54s		master passed
+1 💚	checkstyle	0m 52s		master passed
+1 💚	spotbugs	2m 16s		master passed
+1 💚	spotless	0m 46s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 5s		the patch passed
+1 💚	compile	3m 51s		the patch passed
+1 💚	javac	3m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 54s		the patch passed
+1 💚	spotbugs	2m 31s		the patch passed
+1 💚	hadoopcheck	11m 33s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 43s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 17s		The patch does not generate ASF License warnings.
		42m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5692/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5692
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 0913f5845d1b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `8909436`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5692/6/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-09-23T20:19:07Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 25s		Docker mode activated.
-0 ⚠️	yetus	0m 2s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+0 🆗	mvndep	0m 10s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 15s		master passed
+1 💚	compile	1m 18s		master passed
+1 💚	javadoc	0m 44s		master passed
+1 💚	shadedjars	5m 44s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 6s		the patch passed
+1 💚	compile	1m 18s		the patch passed
+1 💚	javac	1m 18s		the patch passed
+1 💚	javadoc	0m 42s		the patch passed
+1 💚	shadedjars	5m 42s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	1m 36s		hbase-client in the patch passed.
+1 💚	unit	231m 44s		hbase-server in the patch passed.
		260m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5692/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5692
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 0b35c99d6e54 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `8909436`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5692/6/testReport/
Max. process+thread count	4967 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5692/6/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org> Co-authored-by: Peng Lu <lupeng_nwpu@qq.com>

guluo2016 mentioned this pull request Feb 19, 2024
HBASE-28187 NPE when flushing a non-existing column family #5553
Closed

This comment has been minimized.
Sign in to view

ndimiduk requested changes Sep 18, 2024
View reviewed changes

ndimiduk requested review from Apache9 and frostruan September 18, 2024 14:46

guluo2016 force-pushed the hbase_HBASE-28187 branch 2 times, most recently from 58b7960 to 77e0934Compare September 20, 2024 12:00

NPE when flushing a non-existing column family
8909436

guluo2016 force-pushed the hbase_HBASE-28187 branch from 77e0934 to 8909436Compare September 20, 2024 12:02

This comment has been minimized.
Sign in to view

ndimiduk approved these changes Sep 23, 2024
View reviewed changes

This comment has been minimized.
Sign in to view

ndimiduk merged commit 449c446 into apache:masterSep 24, 2024

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 24, 2024
HBASE-28187 NPE when flushing a non-existing column family (apache#5692)
32e3556
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

ndimiduk mentioned this pull request Sep 24, 2024
HBASE-26867 Introduce a FlushProcedure #5256
Merged

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 24, 2024
HBASE-28187 NPE when flushing a non-existing column family (apache#5692)
51b2771
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 24, 2024
HBASE-28187 NPE when flushing a non-existing column family (apache#5692)
7e23a08
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

ndimiduk pushed a commit that referenced this pull request Sep 25, 2024
HBASE-28187NPE when flushing a non-existing column family (#5692)
ac90349
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 25, 2024
HBASE-28187 NPE when flushing a non-existing column family (apache#5692)
bfda2a0
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

ndimiduk pushed a commit to ndimiduk/hbase that referenced this pull request Sep 25, 2024
HBASE-28187 NPE when flushing a non-existing column family (apache#5692)
62244ba
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org>

ndimiduk added a commit that referenced this pull request Sep 26, 2024
HBASE-28187NPE when flushing a non-existing column family (#5692)
6ada554
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org> Co-authored-by: Peng Lu <lupeng_nwpu@qq.com>

ndimiduk added a commit that referenced this pull request Sep 26, 2024
HBASE-28187NPE when flushing a non-existing column family (#5692)
99aa86d
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Hui Ruan <huiruan@apache.org> Co-authored-by: Peng Lu <lupeng_nwpu@qq.com>

guluo2016 deleted the hbase_HBASE-28187 branch October 9, 2024 06:17

Uh oh!

Conversation

guluo2016 commented Feb 19, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ndimidukSep 18, 2024

Choose a reason for hiding this comment

Uh oh!

frostruanSep 18, 2024

Choose a reason for hiding this comment

Uh oh!

ndimidukSep 18, 2024

Choose a reason for hiding this comment

Uh oh!

frostruanSep 18, 2024

Choose a reason for hiding this comment

Uh oh!

ndimidukSep 18, 2024

Choose a reason for hiding this comment

Uh oh!

guluo2016Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

guluo2016 commented Sep 20, 2024

flush 'table', 'non_existing_family'

Flush Procedure is enabled.

Flush Procedure is disabled.

flush 'region_name', 'non_existing_family'

Uh oh!

This comment has been minimized.

This comment has been minimized.

frostruan commented Sep 21, 2024

Uh oh!

ndimiduk left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

Apache-HBase commented Sep 23, 2024

Uh oh!

Apache-HBase commented Sep 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants