Skip to content

HBASE-26217 Clean shared state pollution to avoid flaky tests. - #3615

Open
LALAYANG wants to merge 1 commit into
apache:masterfrom
LALAYANG:anotherFix
Open

HBASE-26217 Clean shared state pollution to avoid flaky tests.#3615
LALAYANG wants to merge 1 commit into
apache:masterfrom
LALAYANG:anotherFix

Conversation

@LALAYANG

@LALAYANGLALAYANG commented Aug 21, 2021•
edited
Loading

Copy link
Copy Markdown

What is the purpose of this PR

  • This PR cleans the state polluted by org.apache.hadoop.hbase.monitoring.TestTaskMonitor.testDoNotPurgeRPCTask.

  • It may be better to clean state pollutions so that some other tests won't fail in the future due to the shared state pollution.

Reproduce the test failure

  • Run the test twice in the same JVM.

Expected result:

  • The tests should run successfully when multiple tests that use this state are run in the same JVM.

Actual result:

  • We get the failure:
    [ERROR] Failures: [ERROR] testDoNotPurgeRPCTask:144 RPC Tasks have been purged! expected:<10> but was:<20>

Why the test fails

  • Each time this test runs,10 RPCtasks are created and added to the ArrayList rpcTasks, but rpcTasks is not completely purged when the test ends. So next time the test starts, the remaining RPC tasks lead to an assertion failure.

Fix

Please let me know which fix you think is better. Thanks!

Issue link: https://issues.apache.org/jira/browse/HBASE-26217

@Apache-HBase

Apache-HBase commented Aug 21, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 1smaster passed
+1 💚compile3m 12smaster passed
+1 💚checkstyle1m 2smaster passed
+1 💚spotbugs2m 4smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 39sthe patch passed
+1 💚compile3m 7sthe patch passed
+1 💚javac3m 7sthe patch passed
+1 💚checkstyle1m 0sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck18m 36sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 15sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
49m 19s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3615
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 07d0d630593a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 6bb5701
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count95 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 21, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 30sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 58smaster passed
+1 💚compile1m 25smaster passed
+1 💚shadedjars9m 6sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 49smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 37sthe patch passed
+1 💚compile1m 21sthe patch passed
+1 💚javac1m 21sthe patch passed
+1 💚shadedjars8m 59spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43sthe patch passed
_ Other Tests _
+1 💚unit143m 1shbase-server in the patch passed.
177m 14s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3615
Optional Testsjavac javadoc unit shadedjars compile
unameLinux acf0638325ed 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9 23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 6bb5701
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count3673 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Aug 21, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 34smaster passed
+1 💚compile1m 10smaster passed
+1 💚shadedjars8m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 39smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 0sthe patch passed
+1 💚compile1m 3sthe patch passed
+1 💚javac1m 3sthe patch passed
+1 💚shadedjars8m 20spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 37sthe patch passed
_ Other Tests _
+1 💚unit149m 8shbase-server in the patch passed.
180m 41s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3615
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b8a2098a94d3 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 6bb5701
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count3458 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@LALAYANGLALAYANG changed the title Clean shared state pollution to avoid flaky tests.HBASE-26217 Clean shared state pollution to avoid flaky tests.Aug 22, 2021
@Apache9

Apache9 commented Aug 22, 2021

Copy link
Copy Markdown
Contributor

Thanks for the PR. It is always good that we have developers want to clean up tests code.

But what I do not understand is, we always create a new TaskMonitor in every test method, so how could it mess up later tests run? Looking at the code in TaskMonitor, we do not have static field in this class?

Thanks.

@LALAYANG

LALAYANG commented Nov 9, 2021

Copy link
Copy Markdown
Author

In test org.apache.hadoop.hbase.monitoring.TestTaskMonitor.testDoNotPurgeRPCTask, TaskMonitor is not created each time, so the pollution is not cleaned.

There are 2 kinds of fixes:

  • Clean all tasks when the test ends.
  • Another solution is to create a new TaskMonitor each time to avoid pollution similar to the other tests (e.g., testTaskMonitorBasics, testTasksGetAbortedOnLeak) in TestTaskMonitor. Create a new TaskMonitor each time to avoid flaky tests. Create a new TaskMonitor each time to avoid flaky tests. LALAYANG/hbase#1
    Please let me know which fix you think is better. Thanks!

@Apache-HBase

Apache-HBase commented Jan 27, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 0sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 35smaster passed
+1 💚compile3m 30smaster passed
+1 💚checkstyle1m 11smaster passed
+1 💚spotbugs2m 17smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 17sthe patch passed
+1 💚compile3m 23sthe patch passed
+1 💚javac3m 23sthe patch passed
+1 💚checkstyle1m 9sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck21m 46sPatch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 💚spotbugs2m 25sthe patch passed
_ Other Tests _
+1 💚asflicense0m 14sThe patch does not generate ASF License warnings.
54m 22s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3615
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux a27fff01d96e 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f6348d4
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 27, 2022

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 42sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 46smaster passed
+1 💚compile1m 16smaster passed
+1 💚shadedjars8m 33sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 41sthe patch passed
+1 💚compile1m 18sthe patch passed
+1 💚javac1m 18sthe patch passed
+1 💚shadedjars8m 26spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42sthe patch passed
_ Other Tests _
-1 ❌unit164m 48shbase-server in the patch failed.
198m 58s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3615
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 82cf4cc56328 4.15.0-161-generic #169-Ubuntu SMP Fri Oct 15 13:41:54 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f6348d4
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count3673 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 27, 2022

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 26sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 59smaster passed
+1 💚compile1m 10smaster passed
+1 💚shadedjars9m 15sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 26sthe patch passed
+1 💚compile1m 8sthe patch passed
+1 💚javac1m 8sthe patch passed
+1 💚shadedjars9m 16spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38sthe patch passed
_ Other Tests _
-1 ❌unit223m 35shbase-server in the patch failed.
258m 27s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3615
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 100be35682ca 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / f6348d4
Default JavaAdoptOpenJDK-1.8.0_282-b08
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/testReport/
Max. process+thread count3790 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3615/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@LALAYANG@Apache-HBase@Apache9