Skip to content

HBASE-27491: do not clear cache on RejectedExecutionException - #4914

Merged
bbeaudreault merged 2 commits into
apache:branch-2from
HubSpot:HBASE-27491_branch-2
Dec 9, 2022
Merged

HBASE-27491: do not clear cache on RejectedExecutionException#4914
bbeaudreault merged 2 commits into
apache:branch-2from
HubSpot:HBASE-27491_branch-2

Conversation

@briaugenreich

@briaugenreichbriaugenreich commented Dec 5, 2022

Copy link
Copy Markdown
Contributor

Batch requests submitted via AsyncProcess can fail if the pool is too small or backing region servers are overloaded. Currently, this will result in a meta cache clear which can further exacerbate the problem. Here we special case the exception to no clear the cache.

cc: @bbeaudreault

@Apache-HBase

Apache-HBase commented Dec 5, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 57sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall2m 21sbranch-2 passed
+1 💚compile0m 16sbranch-2 passed
+1 💚shadedjars3m 54sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 16sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 3sthe patch passed
+1 💚compile0m 17sthe patch passed
+1 💚javac0m 17sthe patch passed
+1 💚shadedjars3m 53spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 15sthe patch passed
_ Other Tests _
+1 💚unit2m 37shbase-client in the patch passed.
17m 53s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#4914
JIRA IssueHBASE-27491
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 713b3121e9d8 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 7b4b4ce
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/testReport/
Max. process+thread count165 (vs. ulimit of 30000)
modulesC: hbase-client U: hbase-client
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 5, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 24sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall3m 31sbranch-2 passed
+1 💚compile0m 23sbranch-2 passed
+1 💚shadedjars4m 42sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 0sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars4m 40spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 18sthe patch passed
_ Other Tests _
+1 💚unit3m 9shbase-client in the patch passed.
23m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4914
JIRA IssueHBASE-27491
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9e7620c40934 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 7b4b4ce
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/testReport/
Max. process+thread count198 (vs. ulimit of 30000)
modulesC: hbase-client U: hbase-client
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 5, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 9sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚mvninstall2m 54sbranch-2 passed
+1 💚compile0m 45sbranch-2 passed
+1 💚checkstyle0m 19sbranch-2 passed
+1 💚spotless0m 44sbranch has no errors when running spotless:check.
+1 💚spotbugs0m 52sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 35sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚checkstyle0m 18sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck12m 48sPatch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.4.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs0m 58sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
26m 17s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4914
JIRA IssueHBASE-27491
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 8c5ce318ba7e 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 7b4b4ce
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count75 (vs. ulimit of 30000)
modulesC: hbase-client U: hbase-client
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault
bbeaudreault reviewed Dec 5, 2022
View reviewed changes
Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java Outdated

cleanServerCache(server, t);
// Do not update cache if exception is from failing to submit action to thread pool
if (!(t instanceof RejectedExecutionException)) {

@bbeaudreaultbbeaudreaultDec 5, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this receiveGlobalFailure method is private and only called in 3 places. Rather than add another two instanceof checks here, thoughts on just adding a boolean argument to the method shouldClearCache? We can pass in false everywhere, except in the place where we already instanceof RejectedExecutionException in sendMultiAction

@Apache-HBase

Apache-HBase commented Dec 6, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 19sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall2m 55sbranch-2 passed
+1 💚compile0m 21sbranch-2 passed
+1 💚shadedjars4m 24sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 19sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 37sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚shadedjars4m 7spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 17sthe patch passed
_ Other Tests _
+1 💚unit2m 50shbase-client in the patch passed.
21m 4s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#4914
JIRA IssueHBASE-27491
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 896bcc92cc05 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / f320ad9
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/testReport/
Max. process+thread count165 (vs. ulimit of 30000)
modulesC: hbase-client U: hbase-client
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 6, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 2sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall3m 35sbranch-2 passed
+1 💚compile0m 24sbranch-2 passed
+1 💚shadedjars4m 40sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 10sthe patch passed
+1 💚compile0m 24sthe patch passed
+1 💚javac0m 24sthe patch passed
+1 💚shadedjars4m 42spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 20sthe patch passed
_ Other Tests _
+1 💚unit3m 19shbase-client in the patch passed.
24m 22s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4914
JIRA IssueHBASE-27491
Optional Testsjavac javadoc unit shadedjars compile
unameLinux ff5b97aca7b6 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / f320ad9
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/testReport/
Max. process+thread count192 (vs. ulimit of 30000)
modulesC: hbase-client U: hbase-client
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 6, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 58sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚mvninstall2m 50sbranch-2 passed
+1 💚compile0m 44sbranch-2 passed
+1 💚checkstyle0m 20sbranch-2 passed
+1 💚spotless0m 43sbranch has no errors when running spotless:check.
+1 💚spotbugs0m 53sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 34sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚checkstyle0m 17sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck12m 55sPatch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.4.
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs0m 58sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
26m 35s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4914
JIRA IssueHBASE-27491
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 1399d7f2cced 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / f320ad9
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count83 (vs. ulimit of 30000)
modulesC: hbase-client U: hbase-client
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4914/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@bbeaudreault
bbeaudreault merged commit d87ab89 into apache:branch-2Dec 9, 2022
bbeaudreault pushed a commit that referenced this pull request Dec 9, 2022
@briaugenreich@bbeaudreault
HBASE-27491Do not clear cache on RejectedExecutionException (#4914)
dd60d7f
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
@bbeaudreault
bbeaudreault deleted the HBASE-27491_branch-2 branch December 9, 2022 13:50
bbeaudreault pushed a commit that referenced this pull request Dec 9, 2022
@briaugenreich@bbeaudreault
HBASE-27491Do not clear cache on RejectedExecutionException (#4914)
f1bfda1
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
bbeaudreault pushed a commit to HubSpot/hbase that referenced this pull request Dec 9, 2022
@briaugenreich@bbeaudreault
HubSpot Backport: HBASE-27491 Do not clear cache on RejectedExecution…
d8db95b
…Exception (apache#4914)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@briaugenreich@petersomogyi
HBASE-27491 Do not clear cache on RejectedExecutionException (apache#…
a1b11f8
…4914)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
(cherry picked from commit f1bfda1)
Change-Id: Ieb307623e33665245a3d81935da6195ac9c17a26
@ndimidukndimiduk mentioned this pull request Jun 3, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bbeaudreaultbbeaudreaultbbeaudreault approved these changes
@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@briaugenreich@Apache-HBase@bbeaudreault@Apache9