Skip to content

HBASE-27340 Artifacts with resolved profiles (#4740) - #4761

Merged
Apache9 merged 1 commit into
apache:masterfrom
saintstack:HBASE-27340.master.branch
Sep 6, 2022
Merged

HBASE-27340 Artifacts with resolved profiles (#4740)#4761
Apache9 merged 1 commit into
apache:masterfrom
saintstack:HBASE-27340.master.branch

Conversation

@saintstack

Copy link
Copy Markdown
Contributor

(Forward port from branch-2; simplified by the fact that there
is no hadoop-2.0 profile on master branch)

Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the profile in the sub-module make it so the flatten
plugin can figure 'hadoop.version' definitively.
(In master there is only the hadoop-3.0 profile).
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.

(Forward port from branch-2; simplified by the fact that there
is no hadoop-2.0 profile on master branch)
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the profile in the sub-module make it so the flatten
plugin can figure 'hadoop.version' definitively.
(In master there is only the hadoop-3.0 profile).
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
@saintstack

Copy link
Copy Markdown
ContributorAuthor

Even though there is no hadoop-2.0 profile in master, there is the hadoop-3.0 profile. A few hadoop dependencies are included but not under the hadoop-3.0 profile and without explicit version which causes flatten-plugin difficultly. This explains movement of dependencies under hadoop-3.0 profiles in this master patch (just as we did on branch-2 but less movement needed).

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 40sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 19sMaven dependency ordering for branch
+1 💚mvninstall2m 18smaster passed
+1 💚compile6m 1smaster passed
+1 💚spotless0m 37sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall2m 2sthe patch passed
+1 💚compile5m 53sthe patch passed
+1 💚javac5m 53sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚xml0m 7sThe patch has no ill-formed XML file.
+1 💚hadoopcheck7m 54sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 35spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense1m 21sThe patch does not generate ASF License warnings.
36m 52s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4761
Optional Testsdupname asflicense javac hadoopcheck spotless xml compile
unameLinux c8fbcb67ef9e 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1fc703f
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count139 (vs. ulimit of 30000)
modulesC: hbase-hadoop-compat hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-endpoint hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util . U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 36sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 20sMaven dependency ordering for branch
+1 💚mvninstall2m 50smaster passed
+1 💚compile1m 52smaster passed
+1 💚shadedjars4m 14sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc3m 58smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall2m 37sthe patch passed
+1 💚compile1m 48sthe patch passed
+1 💚javac1m 48sthe patch passed
+1 💚shadedjars4m 8spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc3m 53sthe patch passed
_ Other Tests _
+1 💚unit250m 37sroot in the patch passed.
283m 41s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4761
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9aad9cc1a7dd 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1fc703f
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/testReport/
Max. process+thread count4613 (vs. ulimit of 30000)
modulesC: hbase-hadoop-compat hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-endpoint hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util . U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 39sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 19sMaven dependency ordering for branch
+1 💚mvninstall2m 23smaster passed
+1 💚compile1m 28smaster passed
+1 💚shadedjars4m 5sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc3m 9smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for patch
+1 💚mvninstall2m 8sthe patch passed
+1 💚compile1m 27sthe patch passed
+1 💚javac1m 27sthe patch passed
+1 💚shadedjars4m 2spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc3m 12sthe patch passed
_ Other Tests _
+1 💚unit391m 34sroot in the patch passed.
421m 39s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4761
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 0e11af349c3d 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1fc703f
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/testReport/
Max. process+thread count4788 (vs. ulimit of 30000)
modulesC: hbase-hadoop-compat hbase-balancer hbase-http hbase-asyncfs hbase-procedure hbase-server hbase-mapreduce hbase-endpoint hbase-rest hbase-examples hbase-shaded/hbase-shaded-testing-util . U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4761/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9
Apache9 merged commit 1b5403c into apache:masterSep 6, 2022
@Apache9

Copy link
Copy Markdown
Contributor

Let's get this done :)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@saintstack@Apache-HBase@Apache9