Skip to content

HBASE-28669: fix leaks a connection to ZooKeeper - #6147

Merged
Apache9 merged 2 commits into
apache:masterfrom
wdzaslzy:HBASE-28669
Sep 4, 2024
Merged

HBASE-28669: fix leaks a connection to ZooKeeper#6147
Apache9 merged 2 commits into
apache:masterfrom
wdzaslzy:HBASE-28669

Conversation

@wdzaslzy

@wdzaslzywdzaslzy commented Aug 9, 2024

Copy link
Copy Markdown
Contributor

@Apache-HBase

Apache-HBase commented Aug 9, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 39sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ master Compile Tests _
+1 💚mvninstall3m 4smaster passed
+1 💚compile2m 56smaster passed
+1 💚checkstyle0m 38smaster passed
+1 💚spotbugs1m 33smaster passed
+1 💚spotless0m 46sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall2m 58sthe patch passed
+1 💚compile2m 53sthe patch passed
+1 💚javac2m 53sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 37sthe patch passed
+1 💚spotbugs1m 38sthe patch passed
+1 💚hadoopcheck10m 41sPatch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚spotless0m 43spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
35m 57s
SubsystemReport/Notes
DockerClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6147
JIRA IssueHBASE-28669
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux 909076c0c3c0 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 83c8878
Default JavaEclipse Adoptium-17.0.11+9
Max. process+thread count84 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/1/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 9, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 26sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 15smaster passed
+1 💚compile0m 56smaster passed
+1 💚javadoc0m 28smaster passed
+1 💚shadedjars5m 49sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall3m 6sthe patch passed
+1 💚compile0m 59sthe patch passed
+1 💚javac0m 59sthe patch passed
+1 💚javadoc0m 27sthe patch passed
+1 💚shadedjars5m 47spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit209m 24shbase-server in the patch passed.
234m 30s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6147
JIRA IssueHBASE-28669
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 3408510806b0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 83c8878
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/1/testReport/
Max. process+thread count5281 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@wdzaslzy

wdzaslzy commented Aug 9, 2024

Copy link
Copy Markdown
ContributorAuthor

When I turn on replication and create a peer, if a RegionServer goes down, after recovery, I delete a peer, but the connection between the peer and zookeeper will not be disconnected.

@wdzaslzy

wdzaslzy commented Aug 9, 2024

Copy link
Copy Markdown
ContributorAuthor

When I checked the code, I found that when RecoveredReplicationSource executed tryFinish, only ReplicationSource was deleted from oldsources, but terminate was not executed, resulting in connection leaks.

Apache9
Apache9 reviewed Aug 12, 2024
View reviewed changes
Comment thread...c/main/java/org/apache/hadoop/hbase/replication/regionserver/RecoveredReplicationSource.java Outdated
() -> {
if (workerThreads.isEmpty()) {
this.getSourceMetrics().clear();
this.terminate("Finished recovering queue");

@Apache9Apache9Aug 12, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here we just need to call replicationEndpoint.stop? The terminate method is designed for terminate a replication source when removing a peer or refreshing a source when there are still data need to be replicated in the source, so it contains a lot of unnecessary logics when we just want to remove a finished recovered source.

@wdzaslzywdzaslzyAug 14, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, calling replicationEndpoint.stop can solve this bug. Don't the workerThreads created during the initialization of RecoveredReplicationSource need to be stopped? I think the context simply removes it from the map.

@Apache9Apache9Aug 14, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReplicationSourceShipper is just a thread, it will quit after we returning from the run method, so we do not need to stop it again. And even if you call terminate method here, there is no way to close these threads as there is a workerThreads.isEmpty check above...

@wdzaslzywdzaslzyAug 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks for your help. It is more reasonable to call replicationEndpoint.stop here without call terminate.

@Apache-HBase

Apache-HBase commented Aug 15, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 39sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ master Compile Tests _
+1 💚mvninstall2m 54smaster passed
+1 💚compile2m 50smaster passed
+1 💚checkstyle0m 37smaster passed
+1 💚spotbugs1m 32smaster passed
+1 💚spotless0m 43sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall2m 56sthe patch passed
+1 💚compile2m 59sthe patch passed
+1 💚javac2m 59sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 38sthe patch passed
+1 💚spotbugs1m 38sthe patch passed
+1 💚hadoopcheck10m 26sPatch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚spotless0m 43spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
35m 28s
SubsystemReport/Notes
DockerClientAPI=1.46 ServerAPI=1.46 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6147
JIRA IssueHBASE-28669
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux e275eb5f7110 5.4.0-186-generic #206-Ubuntu SMP Fri Apr 26 12:31:10 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7df5dc8
Default JavaEclipse Adoptium-17.0.11+9
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/2/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 15, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 0smaster passed
+1 💚compile0m 58smaster passed
+1 💚javadoc0m 26smaster passed
+1 💚shadedjars5m 49sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall3m 5sthe patch passed
+1 💚compile0m 57sthe patch passed
+1 💚javac0m 57sthe patch passed
+1 💚javadoc0m 27sthe patch passed
+1 💚shadedjars5m 45spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit210m 27shbase-server in the patch passed.
235m 29s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6147
JIRA IssueHBASE-28669
Optional Testsjavac javadoc unit compile shadedjars
unameLinux 3513166a8d45 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 7df5dc8
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/2/testReport/
Max. process+thread count5421 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6147/2/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Apache9 merged commit 73651dc into apache:masterSep 4, 2024
Apache9 pushed a commit that referenced this pull request Sep 4, 2024
@wdzaslzyrodenli@Apache9
HBASE-28669After one RegionServer restarts, another RegionServer lea…
1563243
…ks a connection to ZooKeeper (#6147)
Co-authored-by: rodenli <rodenli@tencent.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 73651dc)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wdzaslzy@Apache-HBase@Apache9