Skip to content

HBASE-28037 Replication stuck after switching to new WAL but the queue is empty - #5375

Merged
sunhelly merged 1 commit into
apache:branch-2.4from
sunhelly:HBASE-28037
Sep 28, 2023
Merged

HBASE-28037 Replication stuck after switching to new WAL but the queue is empty#5375
sunhelly merged 1 commit into
apache:branch-2.4from
sunhelly:HBASE-28037

Conversation

@sunhelly

@sunhellysunhelly commented Aug 29, 2023

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Apache-HBase commented Aug 29, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 43sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2.4 Compile Tests _
+1 💚mvninstall2m 39sbranch-2.4 passed
+1 💚compile2m 22sbranch-2.4 passed
+1 💚checkstyle0m 35sbranch-2.4 passed
+1 💚spotless0m 40sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 23sbranch-2.4 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 19sthe patch passed
+1 💚compile2m 20sthe patch passed
+1 💚javac2m 20sthe patch passed
+1 💚checkstyle0m 33sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck17m 7sPatch does not cause any errors with Hadoop 2.10.2 or 3.1.4 3.2.4 3.3.5.
+1 💚spotless0m 39spatch has no errors when running spotless:check.
+1 💚spotbugs1m 29sthe patch passed
_ Other Tests _
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
34m 50s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5375
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 91fefedbb899 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.4 / 61250ad
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
sunhelly requested a review from Apache9August 29, 2023 09:46
@Apache-HBase

Apache-HBase commented Aug 29, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2.4 Compile Tests _
+1 💚mvninstall2m 8sbranch-2.4 passed
+1 💚compile0m 36sbranch-2.4 passed
+1 💚shadedjars3m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sbranch-2.4 passed
_ Patch Compile Tests _
+1 💚mvninstall1m 58sthe patch passed
+1 💚compile0m 35sthe patch passed
+1 💚javac0m 35sthe patch passed
+1 💚shadedjars3m 43spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sthe patch passed
_ Other Tests _
+1 💚unit176m 50shbase-server in the patch passed.
195m 22s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#5375
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 65c71698505e 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.4 / 61250ad
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/testReport/
Max. process+thread count4338 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 29, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 35sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2.4 Compile Tests _
+1 💚mvninstall2m 26sbranch-2.4 passed
+1 💚compile0m 42sbranch-2.4 passed
+1 💚shadedjars4m 10sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sbranch-2.4 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 15sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚shadedjars4m 11spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit176m 14shbase-server in the patch passed.
196m 30s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5375
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 899e8800ebc5 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.4 / 61250ad
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/testReport/
Max. process+thread count4618 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9

Apache9 commented Sep 13, 2023

Copy link
Copy Markdown
Contributor

IIRC it is impossible that a normal replication source has an empty queue since it will always has a wal file being written. Can you reproduce this problem with a UT? Or at least explain the sequence on how to reproduce this problem?

Thanks.

@sunhelly

sunhelly commented Sep 27, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Thanks, @Apache9. There exists replication stuck on our production clusters which will recover after restarting the stuck regionserver. I digged the issue and found that in the jstack info there was no active source stream readers while the replication queue was not empty.
I think the replication log is enqueued by calling WALActionListener#postLogRoll, which means only the wal creation completed will make the wal enqueue. But there are some circumstances, e.g. hardware fails on datanodes, that will make the wal creation operation consums some seconds and with retries. Then when the speed of source reader is fast enough or wals under replicated are short enough, there always exists extremely short-term empty replication queue(especially for wal group, which distinguishes the wal replication queues).
By the way, since when the stream reader is stopped, only when the queue is not exist it will start new reader. Here the reader stopping is not consistent with the group queue REMOVE(NOT EXIST), it only matches the queue EMPTY. I think if we allow the reader stop here, then the logics of restarting the reader should be reconstructed. Only stopping the reader when it is a recovered queue is a safe and simple resolution.

@Apache9

Apache9 commented Sep 27, 2023

Copy link
Copy Markdown
Contributor

Thanks, @Apache9. There exists replication stuck on our production clusters which will recover after restarting the stuck regionserver. I digged the issue and found that in the jstack info there was no active source stream readers while the replication queue was not empty. I think the replication log is enqueued by calling WALActionListener#postLogRoll, which means only the wal creation completed will make the wal enqueue. But there are some circumstances, e.g. hardware fails on datanodes, that will make the wal creation operation consums some seconds and with retries. Then when the speed of source reader is fast enough or wals under replicated are short enough, there always exists extremely short-term empty replication queue(especially for wal group, which distinguishes the wal replication queues). By the way, since when the stream reader is stopped, only when the queue is not exist it will start new reader. Here the reader stopping is not consistent with the group queue REMOVE(NOT EXIST), it only matches the queue EMPTY. I think if we allow the reader stop here, then the logics of restarting the reader should be reconstructed. Only stopping the reader when it is a recovered queue is a safe and simple resolution.

If it is possible that the replication queue could be empty in a very shot time window, then there could be other serious problem, as we do not expect a non recovery replication queue could be empty...
Then we should try to add synchronization or changing the operation order to not allow this happen...
Will take a look at the related code later.

Thanks for reporting.

@Apache9

Apache9 commented Sep 27, 2023

Copy link
Copy Markdown
Contributor

Checked the code, on branch-2.x, we will only record the WAL file on zk in preLogRoll, this is for not losing the WAL after restarting, but we will not enqueue it. The enqueuing is done in postLogRoll. So it is possible that the replication queue is empty for a very short time window.

On master and branch-3, we even do not have preLogRoll implemented any more, only enqueue the log in postLogRoll.

So this is a problem.

I think we can apply this PR for branch-2.5 and branch-2.4.

I will open an issue for handling this problem for other branches, as the code has been refactored a lot...

@Apache9

Apache9 commented Sep 27, 2023

Copy link
Copy Markdown
Contributor

Oh, please change the comments? There is no sync replication for branch-2.x.

@Apache-HBase

Apache-HBase commented Sep 27, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 39sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2.4 Compile Tests _
+1 💚mvninstall3m 46sbranch-2.4 passed
+1 💚compile3m 57sbranch-2.4 passed
+1 💚checkstyle1m 1sbranch-2.4 passed
+1 💚spotless0m 59sbranch has no errors when running spotless:check.
+1 💚spotbugs2m 1sbranch-2.4 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 45sthe patch passed
+1 💚compile3m 40sthe patch passed
+1 💚javac3m 40sthe patch passed
+1 💚checkstyle0m 52sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck24m 14sPatch does not cause any errors with Hadoop 2.10.2 or 3.1.4 3.2.4 3.3.6.
+1 💚spotless1m 4spatch has no errors when running spotless:check.
+1 💚spotbugs2m 32sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
50m 26s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5375
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 08bce4ef6f70 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.4 / ae7dc1d
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 27, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2.4 Compile Tests _
+1 💚mvninstall2m 29sbranch-2.4 passed
+1 💚compile0m 44sbranch-2.4 passed
+1 💚shadedjars4m 20sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sbranch-2.4 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 17sthe patch passed
+1 💚compile0m 43sthe patch passed
+1 💚javac0m 43sthe patch passed
+1 💚shadedjars4m 15spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit176m 23shbase-server in the patch passed.
196m 45s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5375
Optional Testsjavac javadoc unit shadedjars compile
unameLinux e6f01658bae8 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.4 / ae7dc1d
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/testReport/
Max. process+thread count4557 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 27, 2023

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2.4 Compile Tests _
+1 💚mvninstall2m 12sbranch-2.4 passed
+1 💚compile0m 36sbranch-2.4 passed
+1 💚shadedjars4m 0sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sbranch-2.4 passed
_ Patch Compile Tests _
+1 💚mvninstall1m 59sthe patch passed
+1 💚compile0m 36sthe patch passed
+1 💚javac0m 36sthe patch passed
+1 💚shadedjars3m 55spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 22sthe patch passed
_ Other Tests _
+1 💚unit179m 18shbase-server in the patch passed.
198m 3s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#5375
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 258d5e9444e2 5.4.0-156-generic #173-Ubuntu SMP Tue Jul 11 07:25:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2.4 / ae7dc1d
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/testReport/
Max. process+thread count4219 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5375/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
sunhelly merged commit 4c3bffe into apache:branch-2.4Sep 28, 2023
asfgit pushed a commit that referenced this pull request Sep 28, 2023
@sunhelly
HBASE-28037Replication stuck after switching to new WAL but the queu…
a3abc52
…e is empty (#5375)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@sunhelly@petersomogyi
HBASE-28037 Replication stuck after switching to new WAL but the queu…
df07f38
…e is empty (apache#5375)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 4c3bffe)
Change-Id: I4d8b6168ec533c7a5821f4be9d625b1e4f92b21e
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sunhelly@Apache-HBase@Apache9