HBASE-28434 Update assembly to create a tarball with hadoop and witho… by stoty · Pull Request #5766 · apache/hbase

stoty · 2024-03-20T11:28:58Z

…ut hadoop WIP

stoty · 2024-03-20T11:29:58Z

Just a very rough draft of how this could be done.
It's probably better to leave this after the test dependencies have been cleaned up.

ndimiduk · 2024-05-17T14:26:30Z

+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-yarn-api</artifactId>
+ <version>${hadoop-three.version}</version><exclusions>
+ <exclusion>


if these are at scope:provided, is it necessary to also have the exclusions list? Can these exclusions be omitted enitrely?

They probably can.
I wil test it for the next patch version, and remove them if possible.

stoty · 2024-12-09T08:29:20Z

This includes the refactor and the jax-ws changes that have not yet been merged, but otherwise this is my intended "final" version of the assembly changes.

Apache9 · 2024-12-09T08:41:57Z

So we plan to land HBASE-29016 first?

stoty · 2024-12-09T09:37:16Z

Yes.
It is much easier to do the refactor first, most of changes in HBASE-29016 are required for this to work.
I also plan to merge the jax-ws changes first, so that they don't have to be duplicated either.

I haven't merged HBASE-29016 yet to give you and Nick time to review that.

stoty · 2024-12-09T09:38:34Z

@NihalJain also suggested renaming the new dev classpath generation module, I was also waiting if the name is fine by you, or maybe if you have some different suggestion.

stoty · 2024-12-09T19:38:09Z

This is the diff between the original and byo-hadoop assembly lib dir:

https://issues.apache.org/jira/browse/HBASE-28434?focusedCommentId=17904262&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17904262

stoty · 2025-01-09T14:39:07Z

rebased the patch

stoty · 2025-01-16T06:35:42Z

started new CI run

…ut hadoop WIP

Apache-HBase · 2025-01-16T06:58:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	docker	0m 4s		Docker failed to build run-specific yetus/hbase:tp-13752}.

Subsystem	Report/Notes
GITHUB PR	#5766
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5766/9/console
versions	git=2.17.1
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-01-16T07:40:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 27s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
+0 🆗	mvndep	0m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 10s		master passed
+1 💚	compile	7m 53s		master passed
+1 💚	spotless	0m 44s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 51s		the patch passed
+1 💚	compile	7m 54s		the patch passed
+1 💚	javac	7m 54s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shellcheck	0m 2s		No new issues.
+1 💚	xmllint	0m 1s		No new issues.
+1 💚	hadoopcheck	11m 4s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 42s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 28s		The patch does not generate ASF License warnings.
		42m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5766/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5766
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless javac xmllint hadoopcheck compile
uname	Linux 872c620cfd8e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `762d719`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	192 (vs. ulimit of 30000)
modules	C: hbase-assembly hbase-assembly-byo-hadoop . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5766/9/console
versions	git=2.34.1 maven=3.9.8 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2025-01-21T08:52:28Z

+ </dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-server</artifactId>


I think this will pull in bunch of hadoop dependencies? Have you checked the final assembly?

Yes I have (though not after the last rebase ).
That's why I've added all the Hadoop dependencies with provided scope, those will remove those dependencies and their transitive dependencies.
I've added a list of the differences between the assemblies (before the last rebase) to the ticket:
https://issues.apache.org/jira/browse/HBASE-28434?focusedCommentId=17904262&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17904262

Apache9

Tested locally, there are no hadoop jars in the tarball under hbase-assembly-byo-hadoop.

Do we want to publish this tarball in our release too?

stoty · 2025-01-25T15:38:28Z

Thank you.
Yes, I think we should.

stoty · 2025-01-25T15:38:47Z

We need to change the release script for that. right ?

Apache9 · 2025-01-26T14:01:27Z

We need to change the release script for that. right ?

Yes, we need to change the release script, I do not think there are any release manager will do release manually now.

Can file a new issue to do the script change.

stoty · 2025-01-27T05:57:36Z

Opened HBASE-29095 for the relase script change.

…ut hadoop (apache#5766) Signed-off-by: Duo Zhang <zhangduo@apache.org>

…ut hadoop (#5766) Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 93b8d7b)

junegunn · 2025-01-31T04:44:43Z

 fi

+#If Hadoop is not specified with HADOOP_HOME, check that the assembly includes Hadoop
+if [[ -z "${HADOOP_IN_PATH}" && ! -e "lib/hadoop-common*" ]] ; then


@stoty I just noticed that I'm unable to start hbase in the local dev environment (after mvn install) because of this. How about adding this condition?
diff --git a/bin/hbase b/bin/hbase index 8f58bf578a..5f1b845110 100755 --- a/bin/hbase+++ b/bin/hbase@@ -222,7 +222,7 @@ if [ "$HBASE_DISABLE_HADOOP_CLASSPATH_LOOKUP" != "true" ] ; then fi #If Hadoop is not specified with HADOOP_HOME, check that the assembly includes Hadoop -if [[ -z "${HADOOP_IN_PATH}" && ! -e "lib/hadoop-common*" ]] ; then+if [[ $in_dev_env = false && -z "${HADOOP_IN_PATH}" && ! -e "lib/hadoop-common*" ]] ; then echo Installation does not contain Hadoop, and HADOOP_HOME does not point to a Hadoop installation. echo Specify a compatible Hadoop installation via HADOOP_HOME, or use the HBase assembly variant echo that includes Hadoop.

Thanks, opened HBASE-29106

This also breaks the integration test in nightly...
Maybe we should review the condition here?

I think the problem here is that, we should not directly use 'lib/hadoop-common*', since we may execute the script in any directory...
Let's add the ${HBASE_HOME} prefix here... @stoty

Thanks.
Opened #6665 for the fix.

stoty marked this pull request as draft March 20, 2024 11:29

This comment has been minimized.
Sign in to view

ndimiduk self-requested a review May 17, 2024 14:22

ndimiduk reviewed May 17, 2024
View reviewed changes

stoty force-pushed the HBASE-28434 branch from c7b06b5 to 7981178Compare November 20, 2024 12:19

This comment has been minimized.
Sign in to view

stoty force-pushed the HBASE-28434 branch from 7981178 to 7058de3Compare December 9, 2024 08:27

stoty marked this pull request as ready for review December 9, 2024 08:34

stoty requested review from Apache9, NihalJain and ndimiduk December 9, 2024 08:35

This comment has been minimized.
Sign in to view

stoty force-pushed the HBASE-28434 branch from e61f1bc to f4ca972Compare January 9, 2025 14:31

This comment has been minimized.
Sign in to view

HBASE-28434 Update assembly to create a tarball with hadoop and witho…
762d719
…ut hadoop WIP

stoty force-pushed the HBASE-28434 branch from f4ca972 to 762d719Compare January 16, 2025 06:48

Apache9 reviewed Jan 21, 2025
View reviewed changes

Apache9 approved these changes Jan 25, 2025
View reviewed changes

stoty merged commit 93b8d7b into apache:masterJan 27, 2025

stoty added a commit to stoty/hbase that referenced this pull request Jan 27, 2025
HBASE-28434 Update assembly to create a tarball with hadoop and witho…
97d6bbe
…ut hadoop (apache#5766) Signed-off-by: Duo Zhang <zhangduo@apache.org>

stoty mentioned this pull request Jan 27, 2025
HBASE-28434 Update assembly to create a tarball with hadoop and witho… #6635
Closed

stoty added a commit that referenced this pull request Jan 28, 2025
HBASE-28434Update assembly to create a tarball with hadoop and witho…
3875fff
…ut hadoop (#5766) Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 93b8d7b)

junegunn reviewed Jan 31, 2025
View reviewed changes

Uh oh!

Conversation

stoty commented Mar 20, 2024

Uh oh!

stoty commented Mar 20, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

stoty commented Dec 9, 2024

Uh oh!

Apache9 commented Dec 9, 2024

Uh oh!

This comment has been minimized.

stoty commented Dec 9, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stoty commented Dec 9, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty commented Dec 9, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty commented Jan 9, 2025

Uh oh!

stoty commented Jan 16, 2025

Uh oh!

This comment has been minimized.

Apache-HBase commented Jan 16, 2025

Uh oh!

Apache-HBase commented Jan 16, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache9 left a comment

Choose a reason for hiding this comment

Uh oh!

stoty commented Jan 25, 2025

Uh oh!

stoty commented Jan 25, 2025

Uh oh!

Apache9 commented Jan 26, 2025

Uh oh!

stoty commented Jan 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

stoty commented Dec 9, 2024•
edited
Loading

stoty commented Dec 9, 2024•
edited
Loading