Skip to content

HBASE-25489 improve performance of --parse-release-tags - #2867

Merged
ndimiduk merged 1 commit into
apache:masterfrom
ndimiduk:25489-improve-parse-release-tags-perf
Jan 8, 2021
Merged

HBASE-25489 improve performance of --parse-release-tags#2867
ndimiduk merged 1 commit into
apache:masterfrom
ndimiduk:25489-improve-parse-release-tags-perf

Conversation

@ndimiduk

Copy link
Copy Markdown
Member

Profiler shows a lot of time spent in the UPDATE SQL statement. Remove the tight loop and let SQL
do a bulk-update instead.

@ndimiduk

Copy link
Copy Markdown
MemberAuthor

See before and after flame graphs attached to the JIRA.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 30sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 23sMaven dependency ordering for branch
_ Patch Compile Tests _
+0 🆗mvndep0m 7sMaven dependency ordering for patch
_ Other Tests _
1m 57s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2867
Optional Tests
unameLinux 04d52c228b87 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 8f55a2b
Max. process+thread count51 (vs. ulimit of 30000)
modulesC: U:
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 29sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 22sMaven dependency ordering for branch
_ Patch Compile Tests _
+0 🆗mvndep0m 8sMaven dependency ordering for patch
-0 ⚠️pylint0m 12sThe patch generated 1 new + 3 unchanged - 0 fixed = 4 total (was 3)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
_ Other Tests _
+0 🆗asflicense0m 0sASF License check generated no output?
2m 15s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2867
Optional Testsdupname asflicense pylint
unameLinux 7c504442095e 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 8f55a2b
pylinthttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/artifact/yetus-general-check/output/diff-patch-pylint.txt
Max. process+thread count51 (vs. ulimit of 30000)
modulesC: U:
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/console
versionsgit=2.17.1 maven=3.6.3 pylint=2.4.4
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 28sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 21sMaven dependency ordering for branch
_ Patch Compile Tests _
+0 🆗mvndep0m 8sMaven dependency ordering for patch
_ Other Tests _
2m 52s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2867
Optional Tests
unameLinux d593619f11ec 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 8f55a2b
Max. process+thread count50 (vs. ulimit of 30000)
modulesC: U:
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Profiler shows a lot of time spent in the UPDATE SQL statement. Remove the tight loop and let SQL
do a bulk-update instead.
@ndimiduk
ndimidukforce-pushed the 25489-improve-parse-release-tags-perf branch from 26b9f1c to aa358a5CompareJanuary 8, 2021 21:46
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 31sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 7sMaven dependency ordering for branch
_ Patch Compile Tests _
+0 🆗mvndep0m 7sMaven dependency ordering for patch
_ Other Tests _
1m 41s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2867
Optional Tests
unameLinux 21f571f8c1a4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 49aba57
Max. process+thread count46 (vs. ulimit of 30000)
modulesC: U:
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 38sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 8sMaven dependency ordering for branch
_ Patch Compile Tests _
+0 🆗mvndep0m 8sMaven dependency ordering for patch
+1 💚pylint0m 11sThere were no new pylint issues.
+1 💚whitespace0m 0sThe patch has no whitespace issues.
_ Other Tests _
+0 🆗asflicense0m 1sASF License check generated no output?
2m 22s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2867
Optional Testsdupname asflicense pylint
unameLinux 8f4894dd58f7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 49aba57
Max. process+thread count48 (vs. ulimit of 30000)
modulesC: U:
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/2/console
versionsgit=2.17.1 maven=3.6.3 pylint=2.4.4
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 14sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 7sMaven dependency ordering for branch
_ Patch Compile Tests _
+0 🆗mvndep0m 7sMaven dependency ordering for patch
_ Other Tests _
2m 23s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2867
Optional Tests
unameLinux 8489b8542db3 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 49aba57
Max. process+thread count44 (vs. ulimit of 30000)
modulesC: U:
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2867/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@ndimiduk
ndimiduk merged commit 84c4033 into apache:masterJan 8, 2021
@ndimiduk
ndimiduk deleted the 25489-improve-parse-release-tags-perf branch January 8, 2021 22:44
@ndimiduk

Copy link
Copy Markdown
MemberAuthor

Thanks for the reviews.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@ndimiduk@Apache-HBase@saintstack@huaxiangsun