Skip to content

HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 - #5561

Merged
bbeaudreault merged 4 commits into
apache:masterfrom
HubSpot:HBASE-28206
Dec 6, 2023
Merged

HBASE-28206 [JDK17] JVM crashes intermittently on aarch64#5561
bbeaudreault merged 4 commits into
apache:masterfrom
HubSpot:HBASE-28206

Conversation

@bbeaudreault

@bbeaudreaultbbeaudreault commented Dec 5, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

We routinely see JVM crashes originating from our RegionServerMetricsWrapperRunnable in jdk17 on aarch64. Since that method is so large, it's hard to know exactly where or what the problem is. There is an issue submitted, but no indication of whether or when a fix will come.

The core dump indicates its a problem with On Stack Replacement (OSR). Reading up on that, it can kick in when long methods are suddenly deemed hot and attempt to be optimized in place. Taking a stab in the dark based on that, I saw two potential issues in RegionServerMetricsWrapperRunnable:

  1. The method is exceptionally long, especially when considering the loops
  2. There are over 50 volatile variables updated in the parent class

This PR attempts to solve both issues:

  1. Most of the volatile variables are replaced by a single volatile RegionMetricAggregate.
  2. I tried to break down the giant method into organized units:
    a. regionserver-level calculations are left in the run() method
    b. the main aggregate() method of RegionMetricAggregate loops and collects region-level metrics
    c. for each region, an aggregateStores() method is called which loops and collects store-level metrics.

This may not be perfect, but I think it's an improvement for three reasons:

  1. I've been running it internally for a couple weeks and have not seen a crash (previously multiple per day)
  2. It attempts to organize the code a little bit so its easier to manage, and is a net-reduction in code.
  3. If the crash were to resurface, since there are more methods now we'd be able to better pinpoint the problematic frame.

In order to ensure this refactor did not break any metrics, I wrote an exhaustive unit test. It validates each of the aggregated getters, and succeeds against both the old and new implementation.

@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

Apache-HBase commented Dec 5, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 37sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 40smaster passed
+1 💚compile2m 27smaster passed
+1 💚checkstyle0m 34smaster passed
+1 💚spotless0m 40sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 43sthe patch passed
+1 💚compile2m 24sthe patch passed
+1 💚javac2m 24sthe patch passed
-0 ⚠️checkstyle0m 35shbase-server: The patch generated 3 new + 0 unchanged - 1 fixed = 3 total (was 1)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck9m 15sPatch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs1m 37sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
31m 46s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5561
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux ab603117c156 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 25e9228
Default JavaEclipse Adoptium-11.0.17+8
checkstylehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count78 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 5, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 12sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 37smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars4m 50sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 42sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
+1 💚shadedjars4m 49spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sthe patch passed
_ Other Tests _
+1 💚unit220m 48shbase-server in the patch passed.
243m 4s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5561
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 44332dbc22ab 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 25e9228
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/testReport/
Max. process+thread count4819 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 5, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 19smaster passed
+1 💚compile0m 42smaster passed
+1 💚shadedjars4m 48sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 27sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚shadedjars4m 48spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
-1 ❌unit237m 0shbase-server in the patch failed.
258m 36s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5561
Optional Testsjavac javadoc unit shadedjars compile
unameLinux f79662dda915 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 25e9228
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/testReport/
Max. process+thread count4889 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 6, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 38sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 51smaster passed
+1 💚compile2m 29smaster passed
+1 💚checkstyle0m 37smaster passed
+1 💚spotless0m 44sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 33smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 40sthe patch passed
+1 💚compile2m 27sthe patch passed
+1 💚javac2m 27sthe patch passed
+1 💚checkstyle0m 36shbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck10m 20sPatch does not cause any errors with Hadoop 3.2.4 3.3.6.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs1m 38sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
33m 9s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5561
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 539b05235ba0 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 25e9228
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@bbeaudreault
bbeaudreault merged commit 6e421e9 into apache:masterDec 6, 2023
@bbeaudreault
bbeaudreault deleted the HBASE-28206 branch December 6, 2023 17:53
bbeaudreault added a commit that referenced this pull request Dec 6, 2023
@bbeaudreault
HBASE-28206[JDK17] JVM crashes intermittently on aarch64 (#5561)
7f8be79
Signed-off-by: Duo Zhang <zhangduo@apache.org>
bbeaudreault added a commit that referenced this pull request Dec 6, 2023
@bbeaudreault
HBASE-28206[JDK17] JVM crashes intermittently on aarch64 (#5561)
b79ea7d
Signed-off-by: Duo Zhang <zhangduo@apache.org>
bbeaudreault added a commit that referenced this pull request Dec 6, 2023
@bbeaudreault
HBASE-28206[JDK17] JVM crashes intermittently on aarch64 (#5561)
6d7b9c8
Signed-off-by: Duo Zhang <zhangduo@apache.org>
bbeaudreault added a commit that referenced this pull request Dec 6, 2023
@bbeaudreault
HBASE-28206[JDK17] JVM crashes intermittently on aarch64 (#5561)
24ae92f
Signed-off-by: Duo Zhang <zhangduo@apache.org>
bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Dec 6, 2023
@bbeaudreault
HubSpot Backport: HBASE-28206 [JDK17] JVM crashes intermittently on a…
733e3e1
…arch64 (apache#5561)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@Apache-HBase

Apache-HBase commented Dec 6, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 25sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 59smaster passed
+1 💚compile0m 43smaster passed
+1 💚shadedjars5m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 44sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚shadedjars5m 8spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sthe patch passed
_ Other Tests _
-1 ❌unit215m 17shbase-server in the patch failed.
237m 40s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5561
Optional Testsjavac javadoc unit shadedjars compile
unameLinux a676a46a61ed 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 25e9228
Default JavaEclipse Adoptium-11.0.17+8
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/testReport/
Max. process+thread count4734 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Dec 6, 2023

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 39smaster passed
+1 💚compile0m 42smaster passed
+1 💚shadedjars4m 50sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 28sthe patch passed
+1 💚compile0m 41sthe patch passed
+1 💚javac0m 41sthe patch passed
+1 💚shadedjars4m 48spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
-1 ❌unit236m 19shbase-server in the patch failed.
257m 54s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5561
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 074794bd56f4 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 25e9228
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/testReport/
Max. process+thread count4892 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5561/3/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

kadirozde pushed a commit to kadirozde/hbase that referenced this pull request Jan 5, 2024
@bbeaudreault@kadirozde
HBASE-28206 [JDK17] JVM crashes intermittently on aarch64 (apache#5561)
53c582e
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bbeaudreault@Apache-HBase@Apache9