Uh oh!
There was an error while loading. Please reload this page.
HBASE-27340 Artifacts with resolved profiles - #4740
Merged
Merged
Conversation
Apache-HBase
commented
Aug 28, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 28, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
Aug 28, 2022
Apache9
left a comment
Contributor
There was a problem hiding this comment.
The patch looks great.
On master branch, since we only support hadoop 3, I'm not sure whether we should keep the profile and then use the flatten plugin, or we just remove the profile and move the hadoop dependencies to the dependencies section directly.
| <type>test-jar</type> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> |
Contributor
There was a problem hiding this comment.
This is for moving all the hadoop related dependencies to hadoop-2.0 profile or hadoop-3.0 profile?
| </dependencies> | ||
| </profile> | ||
| <!-- | ||
| profile for building against Hadoop 3.0.x. Activate using: |
Contributor
There was a problem hiding this comment.
'Hadoop 3.0.x' -> 'Hadoop 3.x'
| </dependencies> | ||
| </profile> | ||
| <!-- | ||
| profile for building against Hadoop 3.0.0. Activate using: |
ContributorAuthor
There was a problem hiding this comment.
Will fix in next push...
| <plugin> | ||
| <groupId>org.codehaus.mojo</groupId> | ||
| <artifactId>flatten-maven-plugin</artifactId> | ||
| <!--<version>1.3.0</version>--> |
Contributor
There was a problem hiding this comment.
This is because the version has been declared in the parent apache pom?
ContributorAuthor
There was a problem hiding this comment.
Use 'latest'. Will set it to 1.3.0 explicitly next go around.
Contributor
There was a problem hiding this comment.
I have mixed feelings about manually managing plugin versions but agree it is best on balance. In our own internal builds we need to manage things up sometimes and the version property definitions for plugin versions are helpful for that. Other downstreamers will have a similar experience I suspect.
Uh oh!
There was an error while loading. Please reload this page.
Apache-HBase
commented
Aug 28, 2022
💔 -1 overall
This message was automatically generated. |
apurtell
approved these changes
Aug 30, 2022
apurtell
commented
Aug 30, 2022
Contributor
Approved pending clean spotless check and successful test run. Current failures are relevant to this change e.g. |
saintstack
commented
Aug 30, 2022
ContributorAuthor
Let me fix the failure. For master branch, +1 on removing hadoop profiles, etc. but I'd think we'd still want this patch too because hadoop is not only profile and even if no profiles, the published poms are slim w/ runtime-only dependencies w/ explicit versions filled in. |
Apache9
commented
Aug 31, 2022
Contributor
+1 |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
1 similar comment
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
saintstack
commented
Aug 31, 2022
ContributorAuthor
RELEASENOTE changes come of my running spotless locally. |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
1 similar comment
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 1, 2022
💔 -1 overall
This message was automatically generated. |
1 similar comment
Apache-HBase
commented
Sep 1, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 1, 2022
💔 -1 overall
This message was automatically generated. |
Make it so our published poms carry the minimum needed to run an hbase; the published pom has no profiles -- the profiles specified at build time are resolved, their dependencies inlined, and then they are stripped -- and no build-time, or plugins dependencies or properties, etc. Resultant poms have explicit hadoop lib versions baked in -- no more being able to choose hbase with hadoop2 or haddop3 at downstream build time by setting a '-Dhadoop.profile=X.0'. Pattern is to add profiles when none in sub-modules when the flatten plugin complains it can't resolve an hadoop dependency's 'version' (e.g. hadoop-common, hadoop-hdfs). Adding the hadoop-2.0 and hadoop-3.0 profiles in the sub-module make it so the flatten plugin can figure 'hadoop.version' definitively. Another spin on the above happens when profiles already exist in submodule but the flatten plugin is complaining it can't figure figure version on an hadoop dependency NOT under profiles. Below, we move the delinquent hadoop dependency under existing profiles (minikdc was the usual dependency outside profiles in sub-modules that flatten complained about). Sometimes, moving an hadoop dependency under a profile, there would be excludes on the local dependency. If the parent pom excludes section was missing the local excludes, we added them up to the parent module so all excluding is done up there in the parent profile dependencyManagement section. * hbase-asyncfs/pom.xml * hbase-endpoint/pom.xml * hbase-examples/pom.xml * hbase-http/pom.xml * hbase-rest/pom.xml * hbase-server/pom.xml Move the minikdc under profiles so it picks up appropriate hadoop version when the flatten plugin runs. * hbase-hadoop2-compat/pom.xml Add hadoop2 and hadoop3 profiles and move hadoop-common, etc. under them so we pick up appropriate hadoop version when flatten plugin runs. * hbase-mapreduce/pom.xml Move hadoop dependencies under profiles so right version is available when the flatten plugin runs. * hbase-shaded/hbase-shaded-testing-util/pom.xml Add profiles for hadoop-2.0 and hadoop-3.0 and move the hadoop dependencies under them. pom.xml Add the flatten plugin with the flatten profiles enabled. Add a few excludes on hadoop profiles picked up from sub-modules. E.g. exclude bouncycastle bcprov-jdk15 when we include minikdc.
Apache-HBase
commented
Sep 1, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 1, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 1, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
Sep 2, 2022
saintstack added a commit
that referenced
this pull request
Sep 2, 2022
Make it so our published poms carry the minimum needed to run an hbase; the published pom has no profiles -- the profiles specified at build time are resolved, their dependencies inlined, and then they are stripped -- and no build-time, or plugins dependencies or properties, etc. Resultant poms have explicit hadoop lib versions baked in -- no more being able to choose hbase with hadoop2 or haddop3 at downstream build time by setting a '-Dhadoop.profile=X.0'. Pattern is to add profiles when none in sub-modules when the flatten plugin complains it can't resolve an hadoop dependency's 'version' (e.g. hadoop-common, hadoop-hdfs). Adding the hadoop-2.0 and hadoop-3.0 profiles in the sub-module make it so the flatten plugin can figure 'hadoop.version' definitively. Another spin on the above happens when profiles already exist in submodule but the flatten plugin is complaining it can't figure figure version on an hadoop dependency NOT under profiles. Below, we move the delinquent hadoop dependency under existing profiles (minikdc was the usual dependency outside profiles in sub-modules that flatten complained about). Sometimes, moving an hadoop dependency under a profile, there would be excludes on the local dependency. If the parent pom excludes section was missing the local excludes, we added them up to the parent module so all excluding is done up there in the parent profile dependencyManagement section. * hbase-asyncfs/pom.xml * hbase-endpoint/pom.xml * hbase-examples/pom.xml * hbase-http/pom.xml * hbase-rest/pom.xml * hbase-server/pom.xml Move the minikdc under profiles so it picks up appropriate hadoop version when the flatten plugin runs. * hbase-hadoop2-compat/pom.xml Add hadoop2 and hadoop3 profiles and move hadoop-common, etc. under them so we pick up appropriate hadoop version when flatten plugin runs. * hbase-mapreduce/pom.xml Move hadoop dependencies under profiles so right version is available when the flatten plugin runs. * hbase-shaded/hbase-shaded-testing-util/pom.xml Add profiles for hadoop-2.0 and hadoop-3.0 and move the hadoop dependencies under them. pom.xml Add the flatten plugin with the flatten profiles enabled. Add a few excludes on hadoop profiles picked up from sub-modules. E.g. exclude bouncycastle bcprov-jdk15 when we include minikdc. Signed-off-by: Andrew Purtell <apurtell@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>
saintstack added a commit
to saintstack/hbase
that referenced
this pull request
Sep 3, 2022
(Forward port from branch-2; simplified by the fact that there is no hadoop-2.0 profile on master branch) Make it so our published poms carry the minimum needed to run an hbase; the published pom has no profiles -- the profiles specified at build time are resolved, their dependencies inlined, and then they are stripped -- and no build-time, or plugins dependencies or properties, etc. Resultant poms have explicit hadoop lib versions baked in -- no more being able to choose hbase with hadoop2 or haddop3 at downstream build time by setting a '-Dhadoop.profile=X.0'. Pattern is to add profiles when none in sub-modules when the flatten plugin complains it can't resolve an hadoop dependency's 'version' (e.g. hadoop-common, hadoop-hdfs). Adding the profile in the sub-module make it so the flatten plugin can figure 'hadoop.version' definitively. (In master there is only the hadoop-3.0 profile). Another spin on the above happens when profiles already exist in submodule but the flatten plugin is complaining it can't figure figure version on an hadoop dependency NOT under profiles. Below, we move the delinquent hadoop dependency under existing profiles (minikdc was the usual dependency outside profiles in sub-modules that flatten complained about). Sometimes, moving an hadoop dependency under a profile, there would be excludes on the local dependency. If the parent pom excludes section was missing the local excludes, we added them up to the parent module so all excluding is done up there in the parent profile dependencyManagement section.
Apache9 pushed a commit
that referenced
this pull request
Sep 6, 2022
(Forward port from branch-2; simplified by the fact that there is no hadoop-2.0 profile on master branch) Make it so our published poms carry the minimum needed to run an hbase; the published pom has no profiles -- the profiles specified at build time are resolved, their dependencies inlined, and then they are stripped -- and no build-time, or plugins dependencies or properties, etc. Resultant poms have explicit hadoop lib versions baked in -- no more being able to choose hbase with hadoop2 or haddop3 at downstream build time by setting a '-Dhadoop.profile=X.0'. Pattern is to add profiles when none in sub-modules when the flatten plugin complains it can't resolve an hadoop dependency's 'version' (e.g. hadoop-common, hadoop-hdfs). Adding the profile in the sub-module make it so the flatten plugin can figure 'hadoop.version' definitively. (In master there is only the hadoop-3.0 profile). Another spin on the above happens when profiles already exist in submodule but the flatten plugin is complaining it can't figure figure version on an hadoop dependency NOT under profiles. Below, we move the delinquent hadoop dependency under existing profiles (minikdc was the usual dependency outside profiles in sub-modules that flatten complained about). Sometimes, moving an hadoop dependency under a profile, there would be excludes on the local dependency. If the parent pom excludes section was missing the local excludes, we added them up to the parent module so all excluding is done up there in the parent profile dependencyManagement section. Signed-off-by: Duo Zhang <zhangduo@apache.org>
| <id>hadoop-2.0</id> | ||
| <activation> | ||
| <property> | ||
| <!--Below formatting for dev-support/generate-hadoopX-poms.sh--> |
Member
There was a problem hiding this comment.
Heya @saintstack -- this comment is about some support scripts, but I don't see them in the commit. Maybe something you intended to commit with this patch but missed the git add ? Thanks boss.
ContributorAuthor
There was a problem hiding this comment.
Thanks for taking a looksee @ndimiduk
It looks like this is a blind copy/paste of hadoop-2.0 profile activation snippet. I see we are repeating the comment in lots of places (See below). The comment is way stale referring to a script that was removed in 2014.
commit 37654f77a60ad0c13a84d6d6b39657789726df09
Author: Michael Stack <stack@apache.org>
Date: Wed Mar 19 23:08:55 2014 +0000
HBASE-10781 Remove hadoop-one-compat module and all references to hadoop1
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1579449 13f79535-47bb-0310-9956-ffa450edef68
Here is grep for comment over code base:
$ grep -r 'Below formatt' .
./hbase-client/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-asyncfs/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-replication/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-hadoop2-compat/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-zookeeper/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-thrift/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-common/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-examples/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-procedure/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-endpoint/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-testing-util/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-server/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-rest/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-shaded/hbase-shaded-testing-util/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
./hbase-shaded/hbase-shaded-mapreduce/pom.xml: <!--Below formatting for dev-support/generate-hadoopX-poms.sh-->
....
Let me file an issue to clean out the comment... HBASE-27396
Member
There was a problem hiding this comment.
Oh okay. Here I was hoping that there was some additional secret sauce I was missing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make it so our published poms carry the minimum needed to run
an hbase; the published pom has no profiles -- the profiles
specified at build time are resolved, their dependencies inlined,
and then they are stripped -- and no build-time, or plugins
dependencies or properties, etc. Resultant poms have explicit
hadoop lib versions baked in -- no more being able to choose
hbase with hadoop2 or haddop3 at downstream build time by setting
a '-Dhadoop.profile=X.0'.
Pattern is to add profiles when none in sub-modules when
the flatten plugin complains it can't resolve an hadoop
dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
Adding the hadoop-2.0 and hadoop-3.0 profiles in the sub-module
make it so the flatten plugin can figure 'hadoop.version'
definitively.
Another spin on the above happens when profiles already exist
in submodule but the flatten plugin is complaining it can't
figure figure version on an hadoop dependency NOT under
profiles. Below, we move the delinquent hadoop dependency under
existing profiles (minikdc was the usual dependency outside
profiles in sub-modules that flatten complained about).
Sometimes, moving an hadoop dependency under a profile, there
would be excludes on the local dependency. If the parent pom
excludes section was missing the local excludes, we added them
up to the parent module so all excluding is done up there in
the parent profile dependencyManagement section.
hbase-asyncfs/pom.xml
hbase-endpoint/pom.xml
hbase-examples/pom.xml
hbase-http/pom.xml
hbase-rest/pom.xml
hbase-server/pom.xml
Move the minikdc under profiles so it picks up appropriate hadoop version
when the flatten plugin runs.
hbase-hadoop2-compat/pom.xml
Add hadoop2 and hadoop3 profiles and move hadoop-common, etc.
under them so we pick up appropriate hadoop version when flatten
plugin runs.
hbase-mapreduce/pom.xml
Move hadoop dependencies under profiles so right version is
available when the flatten plugin runs.
hbase-shaded/hbase-shaded-testing-util/pom.xml
Add profiles for hadoop-2.0 and hadoop-3.0 and move the
hadoop dependencies under them.
pom.xml
Add the flatten plugin with the flatten profiles enabled.
Add a few excludes on hadoop profiles picked up from sub-modules.
E.g. exclude bouncycastle bcprov-jdk15 when we include minikdc.