Skip to content

HBASE-28921 Avoid bundling hbase-webapps folder in default jars - #6388

Merged
NihalJain merged 3 commits into
apache:masterfrom
NihalJain:HBASE-28921_v2
Nov 4, 2024
Merged

HBASE-28921 Avoid bundling hbase-webapps folder in default jars#6388
NihalJain merged 3 commits into
apache:masterfrom
NihalJain:HBASE-28921_v2

Conversation

@NihalJain

Copy link
Copy Markdown
Contributor

We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.

With this JIRA, we want to avoid bundling static webapp resources in our default jars as these are available during runtime via hbase-webapps directory bundled in our assembly.

But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are copying it to hbase-server tests.

We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are copying it to hbase-server tests.
@NihalJain

NihalJain commented Oct 23, 2024

Copy link
Copy Markdown
ContributorAuthor

2nd attempt for #6368, since we had to revert that one due to #6382

Comment threadpom.xml Outdated
diagnostic info for verifying notice requirements -->
<license.debug.print.included>false</license.debug.print.included>
<!-- When a particular module bundles its depenendencies, should be true -->
<!-- When a particular module bundles its dependencies, should be true -->

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dummy change to trigger all tests

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should revert

@NihalJain

NihalJain commented Oct 23, 2024

Copy link
Copy Markdown
ContributorAuthor

Built hbase locally, untarred and started master, rest and thrift server.
And then verified following:

  • Browse to master, rest and thrift web UIs and ensure no issues in web pages
  • Start hbase shell and do basic CRUD, snapshots etc
  • Run Thrift demo client from hbase-examples
  • Run basic REST APIs
  • Run hbase-diagnostics and hbase-shell UTs locally which helped discover the minicluster issue last time
  • Diff of the tarballs against master:
hbase % jar -tf hbase-server/target/hbase-server-4.0.0-alpha-1-SNAPSHOT.jar | grep hbase-webapps
hbase % jar -tf hbase-server/target/hbase-server-4.0.0-alpha-1-SNAPSHOT-tests.jar | grep hbase-webapps
hbase-webapps/
hbase-webapps/canary/
hbase-webapps/master/
hbase-webapps/master/WEB-INF/
hbase-webapps/regionserver/
hbase-webapps/regionserver/WEB-INF/
hbase-webapps/static/
hbase-webapps/static/css/
hbase-webapps/static/fonts/
hbase-webapps/static/js/
hbase-webapps/canary/index.html
hbase-webapps/master/WEB-INF/web.xml
hbase-webapps/master/index.html
hbase-webapps/regionserver/WEB-INF/web.xml
hbase-webapps/regionserver/index.html
hbase-webapps/static/css/asc.gif
hbase-webapps/static/css/bg.gif
hbase-webapps/static/css/bootstrap-theme.min.css
hbase-webapps/static/css/bootstrap.min.css
hbase-webapps/static/css/desc.gif
hbase-webapps/static/css/hbase.css
hbase-webapps/static/fonts/glyphicons-halflings-regular.eot
hbase-webapps/static/fonts/glyphicons-halflings-regular.svg
hbase-webapps/static/fonts/glyphicons-halflings-regular.ttf
hbase-webapps/static/fonts/glyphicons-halflings-regular.woff
hbase-webapps/static/fonts/glyphicons-halflings-regular.woff2
hbase-webapps/static/hbase_logo.png
hbase-webapps/static/hbase_logo_med.gif
hbase-webapps/static/hbase_logo_small.png
hbase-webapps/static/js/bootstrap.min.js
hbase-webapps/static/js/jqSpager.js
hbase-webapps/static/js/jquery.min.js
hbase-webapps/static/js/jquery.tablesorter.min.js
hbase-webapps/static/js/parser-date-iso8601.min.js
hbase-webapps/static/js/tab.js
hbase-webapps/static/js/vega-embed@6.21.3.min.js
hbase-webapps/static/js/vega-lite@5.6.1.min.js
hbase-webapps/static/js/vega@5.24.0.min.js
hbase-webapps/static/jumping-orca_rotated_12percent.png
hbase % jar -tf hbase-thrift/target/hbase-thrift-4.0.0-alpha-1-SNAPSHOT.jar | grep hbase-webapps
hbase % jar -tf hbase-rest/target/hbase-rest-4.0.0-alpha-1-SNAPSHOT.jar | grep hbase-webapps
hbase % jar -tf hbase-shaded/hbase-shaded-mapreduce/target/hbase-shaded-mapreduce-4.0.0-alpha-1-SNAPSHOT.jar | grep hbase-webapps
hbase % 

CC: @stoty

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

Comment threadhbase-server/pom.xml
@Apache-HBase

This comment has been minimized.

@NihalJain

Copy link
Copy Markdown
ContributorAuthor

Please ignore commit 552c419, will revert before merge. This is to force UTs for all modules

Comment threadhbase-annotations/pom.xml Outdated
</plugins>
</build>
</project>

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should revert at all places

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why bunch of this new line change?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wanted to trigger all tests for this PR, so modified all files to add a dummy whitespace to each pom. Have reverted since build is GREEN now

@NihalJain
NihalJain requested a review from stotyOctober 24, 2024 18:41
@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

Comment threadhbase-server/pom.xml

@stotystoty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

Pending tests and reverting the spurious whitespace changes.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

This comment has been minimized.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec4m 22sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for branch
+1 💚mvninstall3m 1smaster passed
+1 💚compile8m 14smaster passed
+1 💚spotless0m 45sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for patch
+1 💚mvninstall2m 56sthe patch passed
+1 💚compile8m 17sthe patch passed
+1 💚javac8m 17sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚xmllint0m 0sNo new issues.
+1 💚hadoopcheck10m 42sPatch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚spotless0m 44spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 42sThe patch does not generate ASF License warnings.
46m 53s
SubsystemReport/Notes
DockerClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6388/6/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6388
Optional Testsdupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile
unameLinux 8243c127f329 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 434a3ed
Default JavaEclipse Adoptium-17.0.11+9
Max. process+thread count190 (vs. ulimit of 30000)
modulesC: hbase-server hbase-thrift hbase-rest . U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6388/6/console
versionsgit=2.34.1 maven=3.9.8 xmllint=20913
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 35sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+0 🆗mvndep0m 35sMaven dependency ordering for branch
+1 💚mvninstall3m 45smaster passed
+1 💚compile2m 48smaster passed
+1 💚javadoc3m 49smaster passed
+1 💚shadedjars7m 4sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+0 🆗mvndep0m 18sMaven dependency ordering for patch
+1 💚mvninstall3m 54sthe patch passed
+1 💚compile2m 35sthe patch passed
+1 💚javac2m 35sthe patch passed
+1 💚javadoc4m 15sthe patch passed
+1 💚shadedjars6m 57spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit314m 16sroot in the patch passed.
358m 46s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6388/6/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6388
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9a114da4a138 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 434a3ed
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6388/6/testReport/
Max. process+thread count8610 (vs. ulimit of 30000)
modulesC: hbase-server hbase-thrift hbase-rest . U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6388/6/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@NihalJain

Copy link
Copy Markdown
ContributorAuthor

Will wait till EOD to see if any more reviews and then finally merge, in case no more comments here

@NihalJain
NihalJain merged commit 16c51d8 into apache:masterNov 4, 2024
NihalJain added a commit to NihalJain/hbase that referenced this pull request Nov 4, 2024
…he#6388)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit to NihalJain/hbase that referenced this pull request Nov 4, 2024
…he#6388)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit to NihalJain/hbase that referenced this pull request Nov 4, 2024
…he#6388)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit to NihalJain/hbase that referenced this pull request Nov 4, 2024
…he#6388)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit that referenced this pull request Nov 6, 2024
… (#6429)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit that referenced this pull request Nov 6, 2024
… (#6430)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit that referenced this pull request Nov 6, 2024
… (#6431)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
NihalJain added a commit that referenced this pull request Nov 6, 2024
… (#6432)
We are bundling all webapp resources in hbase-server, hbase-thrift, hbase-rest and transitively to hbase-shaded-mapreduce jar. This can be an issue, say if any of the Js projects used by hbase are vulnerable, security scan tools like sonatype start flagging the jars too as vulnerable since they contain vulnerable code.
With this JIRA, we want to avoid bundling static webapp resources in our jars as these are available during runtime via hbase-webapps directory bundled in our assembly.
But, we still need this for our minicluster based tests which expects it to be present in test classpath. Hence, we are copying hbase-webapps to hbase-server tests jar, which contains class SingleProcessHBaseCluster responsible for hbase minicluster creation. This class eventually needs hbase-webapps in classpath during HttpServer initialisation and hence we are adding hbase-webapps to hbase-server test resources.
Signed-off-by: Istvan Toth <stoty@apache.org>
(cherry picked from commit 16c51d8)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@NihalJain@Apache-HBase@stoty@Apache9