Skip to content

PHOENIX-6329 Eliminate or serialize minicluster restart operations in… - #1101

Closed
stoty wants to merge 1 commit into
apache:4.xfrom
stoty:PHOENIX-6329-4.x
Closed

PHOENIX-6329 Eliminate or serialize minicluster restart operations in…#1101
stoty wants to merge 1 commit into
apache:4.xfrom
stoty:PHOENIX-6329-4.x

Conversation

@stoty

Copy link
Copy Markdown
Contributor

… Integration tests

@stoty

Copy link
Copy Markdown
ContributorAuthor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec6m 29sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ 4.x Compile Tests _
+0 🆗mvndep4m 59sMaven dependency ordering for branch
+1 💚mvninstall11m 26s4.x passed
+1 💚compile1m 46s4.x passed
+1 💚checkstyle0m 35s4.x passed
+1 💚javadoc2m 9s4.x passed
+0 🆗spotbugs4m 54sroot in 4.x has 999 extant spotbugs warnings.
+0 🆗spotbugs3m 5sphoenix-core in 4.x has 945 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 19sMaven dependency ordering for patch
-1 ❌mvninstall1m 13sroot in the patch failed.
-1 ❌compile1m 10sroot in the patch failed.
-1 ❌javac1m 10sroot in the patch failed.
+1 💚checkstyle0m 34sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚xml0m 2sThe patch has no ill-formed XML file.
+1 💚javadoc2m 5sthe patch passed
-1 ❌spotbugs1m 3sphoenix-core in the patch failed.
-1 ❌spotbugs1m 30sroot in the patch failed.
_ Other Tests _
-1 ❌unit1m 9sroot in the patch failed.
+1 💚asflicense0m 18sThe patch does not generate ASF License warnings.
45m 50s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1101
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile xml
unameLinux 970e0d57ae6d 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 6fe6052
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
mvninstallhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compilehttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/patch-compile-root.txt
javachttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/patch-compile-root.txt
spotbugshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/patch-spotbugs-phoenix-core.txt
spotbugshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/patch-spotbugs-root.txt
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/artifact/yetus-general-check/output/patch-unit-root.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/testReport/
Max. process+thread count84 (vs. ulimit of 30000)
modulesC: phoenix-core . U: .
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/1/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty
stotyforce-pushed the PHOENIX-6329-4.x branch 3 times, most recently from 6b2e5ad to a1b2ee4CompareJanuary 19, 2021 16:48
@stoty

Copy link
Copy Markdown
ContributorAuthor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec5m 46sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ 4.x Compile Tests _
+0 🆗mvndep5m 5sMaven dependency ordering for branch
+1 💚mvninstall11m 13s4.x passed
+1 💚compile1m 40s4.x passed
+1 💚checkstyle0m 34s4.x passed
+1 💚javadoc2m 0s4.x passed
+0 🆗spotbugs4m 36sroot in 4.x has 999 extant spotbugs warnings.
+0 🆗spotbugs3m 2sphoenix-core in 4.x has 945 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 18sMaven dependency ordering for patch
+1 💚mvninstall6m 55sthe patch passed
+1 💚compile1m 40sthe patch passed
+1 💚javac1m 40sthe patch passed
+1 💚checkstyle0m 34sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚xml0m 2sThe patch has no ill-formed XML file.
+1 💚javadoc2m 2sthe patch passed
+1 💚spotbugs8m 14sthe patch passed
_ Other Tests _
-1 ❌unit200m 33sroot in the patch failed.
+1 💚asflicense0m 20sThe patch does not generate ASF License warnings.
255m 56s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1101
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile xml
unameLinux 8a7e5bd2dfb9 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 4b4a366
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/4/artifact/yetus-general-check/output/patch-unit-root.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/4/testReport/
Max. process+thread count4865 (vs. ulimit of 30000)
modulesC: phoenix-core . U: .
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/4/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@gjacoby126gjacoby126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, assuming that usage data shows this makes the tests more stable. Thanks @stoty

@virajjasani

virajjasani commented Jan 22, 2021

Copy link
Copy Markdown
Contributor

With this, we have perf advantage over #1080 but still full build took 4h16m. I believe a couple more build runs could provide clear results on both PRs.

@stoty

Copy link
Copy Markdown
ContributorAuthor

Comparing the ASF jenkins runtimes is not trivial. We've got both 16 and 24 core machines in the pool, and there may or may not be another task running in parallel.
On my SSD desktop the difference to master was within the margin of error, but I do not have access to proper spinning rust hard disk machines or dedicated VMs to do an apples to apples comparison with HDD I/O.

@stoty

Copy link
Copy Markdown
ContributorAuthor

The last build had a successful test run (the maven shading bug caused a build failure, but that's a different issue)
and the maven test part took 2:02 .
I think that's good enough performance wise, and if we still see a performance regression, we can always work on improving it.

@virajjasani

virajjasani commented Jan 22, 2021

Copy link
Copy Markdown
Contributor

Got it, Thanks. I think it makes sense, anyways when I tried in my VM with and without #1080 , the difference was apparent. Anyways, synchronizing the key functionality of starting and shuting down miniCluster is necessary. Agree with moving forward with this patch after we have QA result for updated PR.

and the maven test part took 2:02

Great.

@stoty

Copy link
Copy Markdown
ContributorAuthor

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 7sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ 4.x Compile Tests _
+0 🆗mvndep5m 0sMaven dependency ordering for branch
+1 💚mvninstall12m 1s4.x passed
+1 💚compile1m 41s4.x passed
+1 💚checkstyle0m 33s4.x passed
+1 💚javadoc2m 5s4.x passed
+0 🆗spotbugs4m 44sroot in 4.x has 999 extant spotbugs warnings.
+0 🆗spotbugs3m 14sphoenix-core in 4.x has 945 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗mvndep0m 19sMaven dependency ordering for patch
+1 💚mvninstall7m 10sthe patch passed
+1 💚compile1m 46sthe patch passed
+1 💚javac1m 46sthe patch passed
+1 💚checkstyle0m 36sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚xml0m 3sThe patch has no ill-formed XML file.
+1 💚javadoc2m 15sthe patch passed
+1 💚spotbugs8m 40sthe patch passed
_ Other Tests _
-1 ❌unit216m 52sroot in the patch failed.
+1 💚asflicense0m 20sThe patch does not generate ASF License warnings.
269m 45s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#1101
Optional Testsdupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile xml
unameLinux 02217a66740e 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev/phoenix-personality.sh
git revision4.x / 49994f8
Default JavaPrivate Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unithttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/6/artifact/yetus-general-check/output/patch-unit-root.txt
Test Resultshttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/6/testReport/
Max. process+thread count4773 (vs. ulimit of 30000)
modulesC: phoenix-core . U: .
Console outputhttps://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1101/6/console
versionsgit=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@stoty
stoty deleted the PHOENIX-6329-4.x branch July 29, 2021 14:14
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@stoty@virajjasani@gjacoby126