Skip to content

HBASE-27720 TestClusterRestartFailover is flakey due to metrics assertion - #5110

Merged
Apache9 merged 1 commit into
apache:masterfrom
ndimiduk:27720-TestClusterRestartFailover
Apr 8, 2023
Merged

HBASE-27720 TestClusterRestartFailover is flakey due to metrics assertion#5110
Apache9 merged 1 commit into
apache:masterfrom
ndimiduk:27720-TestClusterRestartFailover

Conversation

@ndimiduk

Copy link
Copy Markdown
Member

This test has been flakey. When it is rerun by surefire, the underlying minicluster isn't completely cleaned. specifically, the metics system isn't reset. The result is an otherwise successful re-run is failed because there's 8 or 12 SCPcounts instead of the 4 that a single run of the test would otherwise produce. Thus, assert of a multiple of 4.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 47sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall5m 36smaster passed
+1 💚compile3m 6smaster passed
+1 💚checkstyle0m 37smaster passed
+1 💚spotless0m 49sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 45smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 2sthe patch passed
+1 💚compile3m 2sthe patch passed
+1 💚javac3m 2sthe patch passed
+1 💚checkstyle0m 39sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 20sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless1m 3spatch has no errors when running spotless:check.
+1 💚spotbugs2m 24sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
57m 51s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5110
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 4e20ad144ddb 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1f2e1f5
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count84 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 43sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 29smaster passed
+1 💚compile1m 3smaster passed
+1 💚shadedjars5m 15sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 30smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 7sthe patch passed
+1 💚compile0m 58sthe patch passed
+1 💚javac0m 58sthe patch passed
+1 💚shadedjars5m 27spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 28sthe patch passed
_ Other Tests _
+1 💚unit223m 36shbase-server in the patch passed.
254m 21s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5110
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9d23b236d4d2 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1f2e1f5
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/testReport/
Max. process+thread count2448 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 16sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 0smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars4m 18sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 47sthe patch passed
+1 💚compile0m 39sthe patch passed
+1 💚javac0m 39sthe patch passed
+1 💚shadedjars4m 16spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
-1 ❌unit273m 9shbase-server in the patch failed.
294m 33s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5110
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 8ae279f1658b 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1f2e1f5
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/testReport/
Max. process+thread count2264 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/1/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

.getServerNode(SERVER_FOR_TEST);
assertNull("serverNode should be deleted after SCP finished", serverNode);

// this test has been flakey. When it is rerun by surefire, the underlying minicluster isn't

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can not reset the metrics system in the AfterClass method or BeforeClass method?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted a new patch that (I think) will reset the master metrics each time the mini cluster is started.

@Apache9

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/HBASE-14166

Seems added in this issue.

@ndimiduk
ndimidukforce-pushed the 27720-TestClusterRestartFailover branch from 9e49b1f to e82f925CompareMarch 31, 2023 13:27
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 25sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 33smaster passed
+1 💚compile2m 22smaster passed
+1 💚checkstyle0m 34smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 13sthe patch passed
+1 💚compile2m 21sthe patch passed
+1 💚javac2m 21sthe patch passed
+1 💚checkstyle0m 31sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck12m 38sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 52spatch has no errors when running spotless:check.
+1 💚spotbugs2m 1sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
39m 35s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5110
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux 0de3642b6d1c 5.4.0-1094-aws #102~18.04.1-Ubuntu SMP Tue Jan 10 21:07:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 143e9b4
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 2sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 1smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars4m 15sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 49sthe patch passed
+1 💚compile0m 39sthe patch passed
+1 💚javac0m 39sthe patch passed
+1 💚shadedjars4m 18spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
-1 ❌unit214m 16shbase-server in the patch failed.
236m 3s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5110
Optional Testsjavac javadoc unit shadedjars compile
unameLinux c0ba7bb23dec 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 143e9b4
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/testReport/
Max. process+thread count2402 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 59sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 35smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars4m 26sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 24sthe patch passed
+1 💚compile0m 45sthe patch passed
+1 💚javac0m 45sthe patch passed
+1 💚shadedjars4m 24spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit216m 1shbase-server in the patch passed.
240m 36s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5110
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2511f107aa50 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 143e9b4
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/testReport/
Max. process+thread count2457 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/2/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 32smaster passed
+1 💚compile2m 58smaster passed
+1 💚checkstyle0m 40smaster passed
+1 💚spotless0m 49sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 36smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 41sthe patch passed
+1 💚compile2m 55sthe patch passed
+1 💚javac2m 55sthe patch passed
+1 💚checkstyle0m 42sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck15m 45sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 51spatch has no errors when running spotless:check.
+1 💚spotbugs1m 59sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
47m 22s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5110
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux a3f1670ad7df 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 143e9b4
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 19sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 47smaster passed
+1 💚compile0m 40smaster passed
+1 💚shadedjars4m 14sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 49sthe patch passed
+1 💚compile0m 39sthe patch passed
+1 💚javac0m 39sthe patch passed
+1 💚shadedjars4m 16spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit210m 47shbase-server in the patch passed.
231m 27s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5110
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 93c0e2cc8816 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 143e9b4
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/testReport/
Max. process+thread count2639 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 0sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 35smaster passed
+1 💚compile0m 48smaster passed
+1 💚shadedjars4m 25sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 27smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 26sthe patch passed
+1 💚compile0m 46sthe patch passed
+1 💚javac0m 46sthe patch passed
+1 💚shadedjars4m 24spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit216m 8shbase-server in the patch passed.
239m 42s
SubsystemReport/Notes
DockerClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5110
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 290a5f9c0f70 5.4.0-137-generic #154-Ubuntu SMP Thu Jan 5 17:03:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 143e9b4
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/testReport/
Max. process+thread count2618 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5110/3/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Apache9 merged commit f36587c into apache:masterApr 8, 2023
Apache9 pushed a commit that referenced this pull request Apr 8, 2023
…tion (#5110)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit f36587c)
Apache9 pushed a commit that referenced this pull request Apr 8, 2023
…tion (#5110)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit f36587c)
Apache9 pushed a commit that referenced this pull request Apr 8, 2023
…tion (#5110)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit f36587c)
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
…tion (apache#5110)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit f36587c)
(cherry picked from commit 34e05b7)
Change-Id: Ide31e7d4473061e223752c213c6836f5a53a1979
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@ndimiduk@Apache-HBase@Apache9