Skip to content

HBASE-22964 Fix flaky TestClusterRestartFailover and TestClusterResta… - #574

Merged
infraio merged 1 commit into
apache:masterfrom
infraio:HBASE-22964
Sep 8, 2019
Merged

HBASE-22964 Fix flaky TestClusterRestartFailover and TestClusterResta…#574
infraio merged 1 commit into
apache:masterfrom
infraio:HBASE-22964

Conversation

@infraio

@infraioinfraio commented Sep 2, 2019

Copy link
Copy Markdown
Contributor

…rtFailoverSplitWithoutZk

@Apache-HBase

Apache-HBase commented Sep 2, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec47Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall418master passed
+1compile69master passed
+1checkstyle99master passed
+1shadedjars350branch has no errors when building our shaded downstream artifacts.
+1javadoc42master passed
0spotbugs282Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs278master passed
_ Patch Compile Tests _
+1mvninstall383the patch passed
+1compile71the patch passed
+1javac71the patch passed
-1checkstyle95hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1whitespace0The patch has no whitespace issues.
+1shadedjars323patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck1165Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc36the patch passed
+1findbugs257the patch passed
_ Other Tests _
-1unit10357hbase-server in the patch failed.
+1asflicense30The patch does not generate ASF License warnings.
14436
ReasonTests
Failed junit testshadoop.hbase.regionserver.TestHRegionWithInMemoryFlush
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/1/artifact/out/Dockerfile
GITHUB PR#574
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux f0ef73ea245b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-574/out/precommit/personality/provided.sh
git revisionmaster / 8e8bd8b
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/1/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/1/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/1/testReport/
Max. process+thread count4897 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/1/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache9

Apache9 commented Sep 3, 2019

Copy link
Copy Markdown
Contributor

We need to know what is the purpose of the test. The current fix looks like we just give up testing if some conditions are not match...

@infraio

infraio commented Sep 3, 2019

Copy link
Copy Markdown
ContributorAuthor

We need to know what is the purpose of the test. The current fix looks like we just give up testing if some conditions are not match...

It was added by HBASE-21565: Delete dead server from dead server list too early leads to concurrent Server Crash Procedures(SCP) for a same server. So the test purpose is assertFalse(UTIL.getHBaseCluster().getMaster().getServerManager().expireServer(testServer)). It should not submit SCP again if there arleady had a SCP for this server.

@infraio

infraio commented Sep 3, 2019

Copy link
Copy Markdown
ContributorAuthor

Updated the patch. Will test the submit SCP again when serverNode is null.

Apache9
Apache9 requested changes Sep 3, 2019
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still not fully understand this UT here. I guess we need to find a way to make sure that the serverNode is not null?

Comment threadhbase-server/src/test/java/org/apache/hadoop/hbase/master/TestClusterRestartFailover.java Outdated
LOG.info("start to find the procedure of SCP for the severName we choose");
Procedure<?> procedure = UTIL.getHBaseCluster().getMaster().getProcedures().stream().filter(
p -> (p instanceof ServerCrashProcedure) &&
((ServerCrashProcedure) p).getServerName().equals(testServer)).findAny().get();

@Apache9Apache9Sep 3, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional.get will never return null, it will throw a NoSuchElementException...

@infraio

infraio commented Sep 3, 2019

Copy link
Copy Markdown
ContributorAuthor

I still not fully understand this UT here. I guess we need to find a way to make sure that the serverNode is not null?

Checked HBASE-21565. I thought this ut is added to test should submit SCP for a same server twice. Need to mock SCP to stuck it if make sure that serverNode is not null.

@Apache9

Apache9 commented Sep 3, 2019

Copy link
Copy Markdown
Contributor

Checked HBASE-21565. I thought this ut is added to test should submit SCP for a same server twice. Need to mock SCP to stuck it if make sure that serverNode is not null.

Yes, this is possible. You can see TestSCPGetRegionsRace, where we override AssignmentManager and hang the SCP when it calls getRegionsOnServer.

@Apache-HBase

Apache-HBase commented Sep 3, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec39Docker mode activated.
_ Prechecks _
+1dupname0No case conflicting files found.
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
+1test4tests0The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1mvninstall326master passed
+1compile57master passed
+1checkstyle78master passed
+1shadedjars271branch has no errors when building our shaded downstream artifacts.
+1javadoc38master passed
0spotbugs240Used deprecated FindBugs config; considering switching to SpotBugs.
+1findbugs239master passed
_ Patch Compile Tests _
+1mvninstall293the patch passed
+1compile54the patch passed
+1javac54the patch passed
-1checkstyle74hbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1whitespace0The patch has no whitespace issues.
+1shadedjars272patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck937Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1javadoc33the patch passed
+1findbugs244the patch passed
_ Other Tests _
+1unit9396hbase-server in the patch passed.
+1asflicense35The patch does not generate ASF License warnings.
12757
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/2/artifact/out/Dockerfile
GITHUB PR#574
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 184d0bd1364d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-574/out/precommit/personality/provided.sh
git revisionmaster / e50ec76
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/2/artifact/out/diff-checkstyle-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/2/testReport/
Max. process+thread count4374 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/2/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 5, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec0m 0sDocker mode activated.
💔patch2m 14s#574 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
SubsystemReport/Notes
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/3/console
versionsgit=2.17.1
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 6, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec6m 54sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚mvninstall6m 9smaster passed
💚compile1m 9smaster passed
💚checkstyle1m 46smaster passed
💚shadedjars5m 24sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 37smaster passed
💙spotbugs4m 50sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 47smaster passed
_ Patch Compile Tests _
💚mvninstall5m 56sthe patch passed
💚compile0m 57sthe patch passed
💚javac0m 57sthe patch passed
💔checkstyle1m 39shbase-server: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars5m 39spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck22m 50sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 45sthe patch passed
💚findbugs4m 47sthe patch passed
_ Other Tests _
💔unit266m 22shbase-server in the patch failed.
💚asflicense0m 30sThe patch does not generate ASF License warnings.
345m 24s
SubsystemReport/Notes
DockerClient=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/4/artifact/out/Dockerfile
GITHUB PR#574
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 8689ff1d52c4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-574/out/precommit/personality/provided.sh
git revisionmaster / fbd5b5e
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/4/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/4/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/4/testReport/
Max. process+thread count4683 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/4/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 6, 2019

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
💙reexec0m 30sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚mvninstall6m 0smaster passed
💚compile1m 2smaster passed
💚checkstyle1m 33smaster passed
💚shadedjars5m 8sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 39smaster passed
💙spotbugs4m 44sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 42smaster passed
_ Patch Compile Tests _
💚mvninstall5m 29sthe patch passed
💚compile1m 0sthe patch passed
💚javac1m 0sthe patch passed
💚checkstyle1m 29sthe patch passed
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars5m 4spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck20m 11sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 47sthe patch passed
💚findbugs5m 32sthe patch passed
_ Other Tests _
💚unit176m 11shbase-server in the patch passed.
💚asflicense0m 26sThe patch does not generate ASF License warnings.
243m 26s
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/5/artifact/out/Dockerfile
GITHUB PR#574
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 8c5cd2703dc1 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-574/out/precommit/personality/provided.sh
git revisionmaster / 51b6ce3
Default Java1.8.0_181
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/5/testReport/
Max. process+thread count4457 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/5/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 approved these changes Sep 7, 2019
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

@Apache-HBase

Apache-HBase commented Sep 7, 2019

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec1m 6sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 1sThe patch appears to include 2 new or modified test files.
_ master Compile Tests _
💚mvninstall5m 59smaster passed
💚compile0m 58smaster passed
💚checkstyle1m 32smaster passed
💚shadedjars4m 55sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 39smaster passed
💙spotbugs4m 30sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 27smaster passed
_ Patch Compile Tests _
💚mvninstall5m 21sthe patch passed
💚compile0m 58sthe patch passed
💚javac0m 58sthe patch passed
💚checkstyle1m 32sthe patch passed
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars4m 55spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck17m 10sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 34sthe patch passed
💚findbugs4m 35sthe patch passed
_ Other Tests _
💔unit232m 20shbase-server in the patch failed.
💚asflicense0m 27sThe patch does not generate ASF License warnings.
294m 23s
SubsystemReport/Notes
DockerClient=19.03.2 Server=19.03.2 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/6/artifact/out/Dockerfile
GITHUB PR#574
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 2b856c5c70f7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-574/out/precommit/personality/provided.sh
git revisionmaster / 78bae9e
Default Java1.8.0_181
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/6/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/6/testReport/
Max. process+thread count4785 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-574/6/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@infraio
infraio merged commit d8e5c87 into apache:masterSep 8, 2019
@infraio
infraio deleted the HBASE-22964 branch September 8, 2019 10:47
asfgit pushed a commit that referenced this pull request Sep 8, 2019
@infraio
HBASE-22964Fix flaky TestClusterRestartFailover and TestClusterResta…
a4a34e0
…rtFailoverSplitWithoutZk (#574)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit that referenced this pull request Sep 9, 2019
@infraio
HBASE-22964Fix flaky TestClusterRestartFailover and TestClusterResta…
5e98129
…rtFailoverSplitWithoutZk (#574)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
infraio added a commit to infraio/hbase that referenced this pull request Aug 17, 2020
@infraioGuanghao Zhang
HBASE-22964 Fix flaky TestClusterRestartFailover and TestClusterResta…
df39a58
…rtFailoverSplitWithoutZk (apache#574)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
symat pushed a commit to symat/hbase that referenced this pull request Feb 17, 2021
@infraioJenkins
HBASE-22964 Fix flaky TestClusterRestartFailover and TestClusterResta…
8bac4e1
…rtFailoverSplitWithoutZk (apache#574)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 5e98129)
Change-Id: Ib1f0549c24d2b81559e33bc22d4096b246a1e37a
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@infraio@Apache-HBase@Apache9