Skip to content

HBASE-26727 Fix CallDroppedException reporting - #4088

Merged
apurtell merged 1 commit into
apache:masterfrom
HubSpot:HBASE-26727
Feb 19, 2022
Merged

HBASE-26727 Fix CallDroppedException reporting#4088
apurtell merged 1 commit into
apache:masterfrom
HubSpot:HBASE-26727

Conversation

@bbeaudreault

@bbeaudreaultbbeaudreault commented Feb 1, 2022

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Apache-HBase commented Feb 1, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 14sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 57smaster passed
+1 💚compile4m 5smaster passed
+1 💚checkstyle1m 15smaster passed
+1 💚spotbugs2m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 58sthe patch passed
+1 💚compile3m 29sthe patch passed
+1 💚javac3m 29sthe patch passed
-0 ⚠️checkstyle1m 11shbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck26m 38sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs2m 30sthe patch passed
_ Other Tests _
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
63m 17s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4088
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux e1e4ae277976 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 351caa5
Default JavaAdoptOpenJDK-1.8.0_282-b08
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 1, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 30sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 8smaster passed
+1 💚compile1m 22smaster passed
+1 💚shadedjars8m 59sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 41sthe patch passed
+1 💚compile1m 18sthe patch passed
+1 💚javac1m 18sthe patch passed
+1 💚shadedjars8m 40spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43sthe patch passed
_ Other Tests _
+1 💚unit145m 27shbase-server in the patch passed.
179m 49s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4088
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 906ff7c6df9f 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 351caa5
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/testReport/
Max. process+thread count4391 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 1, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 20sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 43smaster passed
+1 💚compile1m 6smaster passed
+1 💚shadedjars9m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 31sthe patch passed
+1 💚compile1m 8sthe patch passed
+1 💚javac1m 8sthe patch passed
+1 💚shadedjars9m 9spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
+1 💚unit222m 5shbase-server in the patch passed.
256m 43s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4088
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 79ce7bf5dd63 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 351caa5
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/testReport/
Max. process+thread count3338 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault
bbeaudreault commented Feb 4, 2022
View reviewed changes
Comment threadhbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/metrics/ExceptionTrackingSource.java
void quotaExceededException();
void rpcThrottlingException();
void callDroppedException();
void callTimedOut();

@bbeaudreaultbbeaudreaultFeb 4, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this in this PR because it's a type of call drop that can happen a lot under excess load. I did not roll it into callDroppedException, because it does not manifest as a CallDroppedException on the client side.

@Apache-HBase

Apache-HBase commented Feb 4, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 10sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 30sMaven dependency ordering for branch
+1 💚mvninstall4m 23smaster passed
+1 💚compile3m 54smaster passed
+1 💚checkstyle1m 23smaster passed
+1 💚spotbugs2m 45smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall4m 18sthe patch passed
+1 💚compile3m 52sthe patch passed
+1 💚javac3m 52sthe patch passed
-0 ⚠️checkstyle1m 11shbase-server: The patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck21m 39sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs3m 9sthe patch passed
_ Other Tests _
+1 💚asflicense0m 22sThe patch does not generate ASF License warnings.
57m 56s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4088
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 61853413284f 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18 11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / e848d3b
Default JavaAdoptOpenJDK-1.8.0_282-b08
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count85 (vs. ulimit of 30000)
modulesC: hbase-hadoop-compat hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 5, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 56sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for branch
+1 💚mvninstall5m 20smaster passed
+1 💚compile1m 43smaster passed
+1 💚shadedjars9m 19sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 2smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall5m 10sthe patch passed
+1 💚compile1m 42sthe patch passed
+1 💚javac1m 42sthe patch passed
+1 💚shadedjars9m 17spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc1m 2sthe patch passed
_ Other Tests _
+1 💚unit0m 47shbase-hadoop-compat in the patch passed.
+1 💚unit205m 28shbase-server in the patch passed.
244m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4088
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 1d503602d69c 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / e848d3b
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/testReport/
Max. process+thread count2512 (vs. ulimit of 30000)
modulesC: hbase-hadoop-compat hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 5, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 25sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for branch
+1 💚mvninstall4m 31smaster passed
+1 💚compile1m 25smaster passed
+1 💚shadedjars9m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 54smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall4m 25sthe patch passed
+1 💚compile1m 24sthe patch passed
+1 💚javac1m 24sthe patch passed
+1 💚shadedjars9m 13spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 55sthe patch passed
_ Other Tests _
+1 💚unit0m 43shbase-hadoop-compat in the patch passed.
+1 💚unit216m 45shbase-server in the patch passed.
253m 17s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4088
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9bdc6d248b11 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / e848d3b
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/testReport/
Max. process+thread count2635 (vs. ulimit of 30000)
modulesC: hbase-hadoop-compat hbase-server U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-4088/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 reviewed Feb 5, 2022
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java
call.setResponse(null, null, CALL_DROPPED_EXCEPTION, "Call dropped, server "
+ (address != null ? address : "(channel closed)") + " is overloaded, please retry.");
call.sendResponseIfReady();
this.rpcServer.getMetrics().exception(CALL_DROPPED_EXCEPTION);

@Apache9Apache9Feb 5, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we call this method in the past? Will this introduce double accounting?

@bbeaudreaultbbeaudreaultFeb 5, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell, we do not report them. We added this exception in https://issues.apache.org/jira/browse/HBASE-26623, but the problem is when we drop a call here it doesn't end up hitting any of the other invocations of this method. So the value is always 0 despite throwing these exceptions to clients.

@bbeaudreaultbbeaudreaultFeb 5, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically this is called in a few places, for example in RpcServer or in a few try/catches in RSRpcServices. But a couple exceptions need to be handled a level above this. For example CallQueueTooBigException is handled in the connection and RequestTooBig in the netty frame decoder.

So this exception is another example of a special case like CallQueueTooBig/RequestTooBig, that doesn't get handled by the default try/catches we have in RpcServer/RSRpcServices.

@bbeaudreault

bbeaudreault commented Feb 13, 2022

Copy link
Copy Markdown
ContributorAuthor

@Apache9 any other thoughts here? Would love to get it merged, it's been working ok for us.

apurtell
apurtell approved these changes Feb 16, 2022
View reviewed changes

@apurtellapurtell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Apache9 I assume you have no objection to merging but please lmk if you do. Otherwise will merge this tomorrow.

@apurtell
apurtell merged commit a49c758 into apache:masterFeb 19, 2022
asfgit pushed a commit that referenced this pull request Feb 19, 2022
@bbeaudreault@apurtell
HBASE-26727Fix CallDroppedException reporting (#4088)
2a7b413
Signed-off-by: Andrew Purtell <apurtell@apache.org>
asfgit pushed a commit that referenced this pull request Feb 19, 2022
@bbeaudreault@apurtell
HBASE-26727Fix CallDroppedException reporting (#4088)
1fb4ee3
Signed-off-by: Andrew Purtell <apurtell@apache.org>
@charlesconnell
charlesconnell deleted the HBASE-26727 branch September 19, 2025 15:18
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 left review comments
@apurtellapurtellapurtell approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bbeaudreault@Apache-HBase@apurtell@Apache9