Skip to content

HBASE-27857 Fix timeout exception handling in HBaseClassTestRule - #5231

Merged
Apache9 merged 1 commit into
apache:masterfrom
linux-on-ibm-z:hbaseclasstestrule-timeout-exception
May 13, 2023
Merged

HBASE-27857 Fix timeout exception handling in HBaseClassTestRule#5231
Apache9 merged 1 commit into
apache:masterfrom
linux-on-ibm-z:hbaseclasstestrule-timeout-exception

Conversation

@jonathan-albrecht-ibm

Copy link
Copy Markdown
Contributor

HBaseClassTestRule applies a timeout and a system exit rule to tests. The timeout rule throws an exception if it hits the timeout threshold. Since the timeout rule is applied after the system exit rule, the system exit rule does not see the exception and does not re-enable the system exit behavior which can cause maven to hang on some tests.

This change applies the timeout rule before the system exit rule so that normal system exit can be restored before the surefire forked node is shutdown.

HBaseClassTestRule applies a timeout and a system exit rule to tests.
The timeout rule throws an exception if it hits the timeout threshold.
Since the timeout rule is applied after the system exit rule, the
system exit rule does not see the exception and does not re-enable
the system exit behavior which can cause maven to hang on some
tests.
This change applies the timeout rule before the system exit rule so
that normal system exit can be restored before the surefire forked
node is shutdown.
Signed-off-by: Jonathan Albrecht <jonathan.albrecht@ibm.com>
@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 22sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
-1 ❌mvninstall0m 17sroot in master failed.
-1 ❌compile0m 11shbase-common in master failed.
-1 ❌shadedjars0m 12sbranch has 7 errors when building our shaded downstream artifacts.
-0 ⚠️javadoc0m 7shbase-common in master failed.
_ Patch Compile Tests _
+1 💚mvninstall3m 7sthe patch passed
+1 💚compile0m 16sthe patch passed
+1 💚javac0m 16sthe patch passed
+1 💚shadedjars4m 49spatch has no errors when building our shaded downstream artifacts.
-0 ⚠️javadoc0m 15shbase-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
_ Other Tests _
+1 💚unit1m 49shbase-common in the patch passed.
12m 41s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5231
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 1e83cb88cae7 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5cea811
Default JavaTemurin-1.8.0_352-b08
mvninstallhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk8-hadoop3-check/output/branch-mvninstall-root.txt
compilehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk8-hadoop3-check/output/branch-compile-hbase-common.txt
shadedjarshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk8-hadoop3-check/output/branch-shadedjars.txt
javadochttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk8-hadoop3-check/output/branch-javadoc-hbase-common.txt
javadochttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk8-hadoop3-check/output/diff-javadoc-javadoc-hbase-common.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/testReport/
Max. process+thread count162 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 23sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 35smaster passed
+1 💚compile0m 15smaster passed
+1 💚shadedjars4m 37sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 15smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 17sthe patch passed
+1 💚compile0m 15sthe patch passed
+1 💚javac0m 15sthe patch passed
+1 💚shadedjars4m 34spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 12sthe patch passed
_ Other Tests _
+1 💚unit2m 24shbase-common in the patch passed.
21m 7s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5231
Optional Testsjavac javadoc unit shadedjars compile
unameLinux a1b4edb118ac 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5cea811
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/testReport/
Max. process+thread count192 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall5m 16smaster passed
+1 💚compile0m 50smaster passed
+1 💚checkstyle0m 23smaster passed
+1 💚spotless1m 6sbranch has no errors when running spotless:check.
+1 💚spotbugs0m 46smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 9sthe patch passed
+1 💚compile0m 45sthe patch passed
+1 💚javac0m 45sthe patch passed
+1 💚checkstyle0m 20sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 1sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 58spatch has no errors when running spotless:check.
+1 💚spotbugs0m 53sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
47m 12s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5231
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 7f65005089fd 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 5cea811
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count85 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9

Copy link
Copy Markdown
Contributor

Mind explain a bit more? IIRC, the SystemExitRule is to forbid System.exit call, so what is actual problem here?

@jonathan-albrecht-ibm

Copy link
Copy Markdown
ContributorAuthor

Mind explain a bit more? IIRC, the SystemExitRule is to forbid System.exit call, so what is actual problem here?

Yes, the SystemExitRule is supposed to forbid the System.exit call while a test is running so that the various test servers running in the surefire forked jvm don't cause it to exit before a test has finished IIUC. The SystemExitRule will restore the normal System.exit behaviour when the test completes. The problem is, if a test times out, the test doesn't actually complete. The forked jvm really does need to exit but the SystemExitRule hasn't restored System.exit so it can't.

Note that the timeout exception originates in the timeout rule part of HBaseClassTestRule. The timeout rule actually runs the test method in a separate thread in a Future with a timeout value.

The change in this PR applies the systemExitRule first so when the timeout rule throws an exception it will go through the try...finally block in the systemExitRule which restores the normal System.exit and the forked jvm can exit normally when it needs to.

Without this change, you can see that org.apache.hadoop.hbase.TestSecurityManager shows up in the stacktrace if a test times out:

Created at 2023-04-27T15:51:58.947
org.apache.hadoop.hbase.SystemExitRule$SystemExitInTestException
at org.apache.hadoop.hbase.TestSecurityManager.checkExit(TestSecurityManager.java:32)
at java.base/java.lang.Runtime.exit(Runtime.java:114)
at java.base/java.lang.System.exit(System.java:1752)
at org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:381)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:178)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 22sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 48smaster passed
+1 💚compile0m 14smaster passed
+1 💚shadedjars4m 38sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 12smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 48sthe patch passed
+1 💚compile0m 14sthe patch passed
+1 💚javac0m 14sthe patch passed
+1 💚shadedjars4m 39spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 11sthe patch passed
_ Other Tests _
+1 💚unit1m 41shbase-common in the patch passed.
18m 48s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5231
Optional Testsjavac javadoc unit shadedjars compile
unameLinux dbfeed27a172 5.4.0-1099-aws #107~18.04.1-Ubuntu SMP Fri Mar 17 16:49:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / e343584
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/testReport/
Max. process+thread count163 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 26sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 39smaster passed
+1 💚compile0m 14smaster passed
+1 💚shadedjars4m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 15smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 22sthe patch passed
+1 💚compile0m 15sthe patch passed
+1 💚javac0m 15sthe patch passed
+1 💚shadedjars4m 47spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 13sthe patch passed
_ Other Tests _
+1 💚unit2m 27shbase-common in the patch passed.
21m 29s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5231
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 10589cff1b86 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / e343584
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/testReport/
Max. process+thread count199 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 50sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 16smaster passed
+1 💚compile0m 32smaster passed
+1 💚checkstyle0m 15smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs0m 32smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 12sthe patch passed
+1 💚compile0m 32sthe patch passed
+1 💚javac0m 32sthe patch passed
+1 💚checkstyle0m 14sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck12m 1sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs0m 37sthe patch passed
_ Other Tests _
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
30m 28s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5231
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux cdc6d774ffa7 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / e343584
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count80 (vs. ulimit of 30000)
modulesC: hbase-common U: hbase-common
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5231/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Apache9 merged commit 220eacf into apache:masterMay 13, 2023
Apache9 pushed a commit that referenced this pull request May 13, 2023
)
HBaseClassTestRule applies a timeout and a system exit rule to tests.
The timeout rule throws an exception if it hits the timeout threshold.
Since the timeout rule is applied after the system exit rule, the
system exit rule does not see the exception and does not re-enable
the system exit behavior which can cause maven to hang on some
tests.
This change applies the timeout rule before the system exit rule so
that normal system exit can be restored before the surefire forked
node is shutdown.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 220eacf)
Apache9 pushed a commit that referenced this pull request May 13, 2023
)
HBaseClassTestRule applies a timeout and a system exit rule to tests.
The timeout rule throws an exception if it hits the timeout threshold.
Since the timeout rule is applied after the system exit rule, the
system exit rule does not see the exception and does not re-enable
the system exit behavior which can cause maven to hang on some
tests.
This change applies the timeout rule before the system exit rule so
that normal system exit can be restored before the surefire forked
node is shutdown.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 220eacf)
Apache9 pushed a commit that referenced this pull request May 13, 2023
)
HBaseClassTestRule applies a timeout and a system exit rule to tests.
The timeout rule throws an exception if it hits the timeout threshold.
Since the timeout rule is applied after the system exit rule, the
system exit rule does not see the exception and does not re-enable
the system exit behavior which can cause maven to hang on some
tests.
This change applies the timeout rule before the system exit rule so
that normal system exit can be restored before the surefire forked
node is shutdown.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 220eacf)
@jonathan-albrecht-ibm

Copy link
Copy Markdown
ContributorAuthor

Thanks for reviewing and merging @Apache9!

vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
…ache#5231)
HBaseClassTestRule applies a timeout and a system exit rule to tests.
The timeout rule throws an exception if it hits the timeout threshold.
Since the timeout rule is applied after the system exit rule, the
system exit rule does not see the exception and does not re-enable
the system exit behavior which can cause maven to hang on some
tests.
This change applies the timeout rule before the system exit rule so
that normal system exit can be restored before the surefire forked
node is shutdown.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 220eacf)
(cherry picked from commit 4df157d)
Change-Id: Id051ff2cf38fcdd3ef47f7e8d9ec99a8de88b6c8
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jonathan-albrecht-ibm@Apache-HBase@Apache9