HBASE-27562 Publish SBOM artifacts by dongjoon-hyun · Pull Request #4953 · apache/hbase

dongjoon-hyun · 2023-01-10T02:31:26Z

This PR aims to publish SBOM artifacts along with the other Apache projects.

https://cwiki.apache.org/confluence/display/COMDEV/SBOM

Here is an article to give some context.

https://www.activestate.com/blog/why-the-us-government-is-mandating-software-bill-of-materials-sbom/

Software Bill of Materials (SBOM) are additional artifacts containing the aggregate of all direct and transitive dependencies of a project. The US Government (based on NIST recommendations) currently accepts only the three most popular SBOM standards as valid, namely: CycloneDX, Software Identification (SWID) tag, Software Package Data Exchange® (SPDX).

We can use one of the Maven plugin, CycloneDX maven plugin, a lightweight software bill of materials (SBOM) standard designed for use in application security contexts and supply chain component analysis.

https://maven.apache.org/plugins/index.html#misc

The expected results

$ mvn install -DskipTests
...
$ ls -al ~/.m2/repository/org/apache/hbase/hbase-common/3.0.0-alpha-4-SNAPSHOT
total 5064
drwxr-xr-x 11 dongjoon staff 352 Jan 9 18:28 .
drwxr-xr-x 4 dongjoon staff 128 Jan 9 18:28 ..
-rw-r--r-- 1 dongjoon staff 482 Jan 9 18:28 _remote.repositories
-rw-r--r-- 1 dongjoon staff 159174 Jan 9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-cyclonedx.json
-rw-r--r-- 1 dongjoon staff 139170 Jan 9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-cyclonedx.xml
-rw-r--r-- 1 dongjoon staff 684842 Jan 9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-sources.jar
-rw-r--r-- 1 dongjoon staff 267751 Jan 9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-test-sources.jar
-rw-r--r-- 1 dongjoon staff 443154 Jan 9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT-tests.jar
-rw-r--r-- 1 dongjoon staff 871542 Jan 9 18:28 hbase-common-3.0.0-alpha-4-SNAPSHOT.jar
-rw-r--r-- 1 dongjoon staff 6620 Jan 9 18:27 hbase-common-3.0.0-alpha-4-SNAPSHOT.pom
-rw-r--r-- 1 dongjoon staff 1811 Jan 9 18:28 maven-metadata-local.xml

dongjoon-hyun · 2023-01-10T02:33:21Z

Could you review this, @ndimiduk , @huaxiangsun ?

Apache-HBase · 2023-01-10T03:45:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 5s	master passed
+1 💚	compile	5m 55s	master passed
+1 💚	spotless	0m 59s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 20s	the patch passed
+1 💚	compile	5m 53s	the patch passed
+1 💚	javac	5m 53s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	19m 57s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌	spotless	0m 13s	patch has 22 errors when running spotless:check, run spotless:apply to fix.
		_ Other Tests _
+1 💚	asflicense	0m 20s	The patch does not generate ASF License warnings.
		59m 47s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4953
Optional Tests	dupname asflicense javac hadoopcheck spotless xml compile
uname	Linux d4c7d0da981e 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3f1087f`
Default Java	Eclipse Adoptium-11.0.17+8
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count	81 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-01-10T07:30:37Z

Please run mvn spotless:apply to fix the format error?

Apache-HBase · 2023-01-10T08:43:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 23s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 56s	master passed
+1 💚	compile	2m 20s	master passed
+1 💚	shadedjars	4m 34s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 39s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 57s	the patch passed
+1 💚	compile	2m 14s	the patch passed
+1 💚	javac	2m 14s	the patch passed
+1 💚	shadedjars	7m 4s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	3m 17s	the patch passed
		_ Other Tests _
-1 ❌	unit	322m 19s	root in the patch failed.
		358m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4953
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 3087f985030f 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3f1087f`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/testReport/
Max. process+thread count	2434 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-01-10T09:51:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 14s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 37s	master passed
+1 💚	compile	2m 3s	master passed
+1 💚	shadedjars	4m 54s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 1s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 16s	the patch passed
+1 💚	compile	2m 12s	the patch passed
+1 💚	javac	2m 12s	the patch passed
+1 💚	shadedjars	5m 27s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 38s	the patch passed
		_ Other Tests _
+1 💚	unit	388m 9s	root in the patch passed.
		426m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4953
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9df1f3b4c312 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `3f1087f`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/testReport/
Max. process+thread count	4761 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

dongjoon-hyun · 2023-01-10T09:53:43Z

Thank you, @Apache9 . I did mvn spotless:apply and applied it now.

Apache-HBase · 2023-01-10T12:07:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 54s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	4m 19s	master passed
+1 💚	compile	6m 7s	master passed
+1 💚	spotless	0m 50s	branch has no errors when running spotless:check.
		_ Patch Compile Tests _
+1 💚	mvninstall	5m 49s	the patch passed
+1 💚	compile	6m 12s	the patch passed
+1 💚	javac	6m 12s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 1s	The patch has no ill-formed XML file.
+1 💚	hadoopcheck	19m 58s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	1m 5s	patch has no errors when running spotless:check.
		_ Other Tests _
+1 💚	asflicense	0m 19s	The patch does not generate ASF License warnings.
		57m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4953
Optional Tests	dupname asflicense javac hadoopcheck spotless xml compile
uname	Linux 0fc75f598802 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4add525`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2023-01-10T12:48:17Z

FYI @dongjoon-hyun most of HBase's dependencies are masked behind the hbase-thirdparty libraries. For full utility, I suspect that we'll need that project to also publish sbom.

ndimiduk · 2023-01-10T12:58:05Z

Okay this is a problem.

[WARNING] The following plugins are not marked as thread-safe in Apache HBase:
[WARNING] org.cyclonedx:cyclonedx-maven-plugin:2.7.3
[WARNING] [WARNING] Enable debug to see precisely which goals are not marked as thread-safe.

Is there a newer version of this plugin that supports multi-threaded builds?

ndimiduk · 2023-01-10T12:59:33Z

Looks like not yet, CycloneDX/cyclonedx-maven-plugin#77

ndimiduk · 2023-01-10T13:14:39Z

Have you explored the Apache release process for the resulting artifacts. Does the release manager need to do anything special with the attached artifacts in order to publish them properly?

Apache-HBase · 2023-01-10T16:54:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 28s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 19s	master passed
+1 💚	compile	2m 20s	master passed
+1 💚	shadedjars	4m 30s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 12s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 1s	the patch passed
+1 💚	compile	2m 14s	the patch passed
+1 💚	javac	2m 14s	the patch passed
+1 💚	shadedjars	5m 6s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	311m 10s	root in the patch passed.
		343m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4953
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c44657c7ea78 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4add525`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/testReport/
Max. process+thread count	4866 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

dongjoon-hyun · 2023-01-10T16:54:55Z

Thank you for review, @ndimiduk . This activity focuses on publishing SBOMs of the Maven Central jars.
So, the SBOMs are also published into Maven Central as an immutable files and signed by the ASF signer in the same way with jars. FYI, here is Apache ORC 1.8.2 RC1 vote artifact which is I'm leading now, @ndimiduk .

https://repository.apache.org/content/repositories/orgapacheorc-1064/org/apache/orc/orc-core/1.8.2/

Apache-HBase · 2023-01-10T18:22:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 14s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 32s	master passed
+1 💚	compile	2m 16s	master passed
+1 💚	shadedjars	4m 49s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	2m 20s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 21s	the patch passed
+1 💚	compile	2m 5s	the patch passed
+1 💚	javac	2m 5s	the patch passed
+1 💚	shadedjars	5m 32s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 40s	the patch passed
		_ Other Tests _
+1 💚	unit	397m 42s	root in the patch passed.
		431m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4953
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b9a038f125a0 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `4add525`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/testReport/
Max. process+thread count	4788 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4953/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2023-01-18T13:41:09Z

Have you explored the Apache release process for the resulting artifacts. Does the release manager need to do anything special with the attached artifacts in order to publish them properly?

Let me test locally.

Apache9 · 2023-01-18T14:20:29Z

This is what I've gotten in the local staging dir by running command

mvn clean deploy -DskipTests -Dcheckstyle.skip=true -DaltStagingDirectory=/home/zhangduo/sbom-staged -P apache-release,release -DskipRemoteStaging

zhangduo@zhangduo-VirtualBox:~/sbom-staged/deferred/org/apache/hbase/hbase-client/3.0.0-alpha-4-SNAPSHOT$ ll -h hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.*
-rw-rw-r-- 1 zhangduo zhangduo 229K 1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.json
-rw-rw-r-- 1 zhangduo zhangduo 833 1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.json.asc
-rw-rw-r-- 1 zhangduo zhangduo 196K 1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.xml
-rw-rw-r-- 1 zhangduo zhangduo 833 1月 18 21:53 hbase-client-3.0.0-alpha-4-SNAPSHOT-cyclonedx.xml.asc

Seems fine, we will publish the sbom files along with other files, no more works needed.

And as @ndimiduk have already pointed out, for hbase-thirdparty there is a problem that, we shade and relocate other libraries so we will miss the information when others depend on hbase-thirdparty. But anyway, I think this can be improved later.

dongjoon-hyun · 2023-01-18T17:07:39Z

Thank you so much, @Apache9!

Apache9 · 2023-01-19T07:12:43Z

Going to merge this later unless objections.

Thanks.

Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 5feb06b)

dongjoon-hyun · 2023-01-20T09:48:48Z

Thank you all!

Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 5feb06b) (cherry picked from commit 67a30f3) Change-Id: I164dea1e48ad1ec524f867547d454d4555984549

HBASE-27562Publish SBOM artifacts
a86fe42

Fix
24caea9

ndimiduk requested review from busbey and joshelser January 10, 2023 12:43

Apache9 approved these changes Jan 18, 2023
View reviewed changes

Apache9 merged commit 5feb06b into apache:masterJan 20, 2023

Apache9 pushed a commit that referenced this pull request Jan 20, 2023
HBASE-27562Publish SBOM artifacts (#4953)
af9da14
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 5feb06b)

Apache9 pushed a commit that referenced this pull request Jan 20, 2023
HBASE-27562Publish SBOM artifacts (#4953)
fae5c28
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 5feb06b)

Apache9 pushed a commit that referenced this pull request Jan 20, 2023
HBASE-27562Publish SBOM artifacts (#4953)
67a30f3
Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit 5feb06b)

dongjoon-hyun deleted the HBASE-27562 branch January 20, 2023 09:48

lfrancke mentioned this pull request Feb 8, 2023
Create and publish Software Bill of Materials (SBOMs) for our operators, products and docker images stackabletech/issues#168
Closed

Uh oh!

Conversation

dongjoon-hyun commented Jan 10, 2023

Uh oh!

dongjoon-hyun commented Jan 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Jan 10, 2023

Uh oh!

Apache9 commented Jan 10, 2023

Uh oh!

Apache-HBase commented Jan 10, 2023

Uh oh!

Apache-HBase commented Jan 10, 2023

Uh oh!

dongjoon-hyun commented Jan 10, 2023

Uh oh!

Apache-HBase commented Jan 10, 2023

Uh oh!

ndimiduk commented Jan 10, 2023

Uh oh!

ndimiduk commented Jan 10, 2023

Uh oh!

ndimiduk commented Jan 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ndimiduk commented Jan 10, 2023

Uh oh!

Apache-HBase commented Jan 10, 2023

Uh oh!

dongjoon-hyun commented Jan 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Jan 10, 2023

Uh oh!

Apache9 commented Jan 18, 2023

Uh oh!

Apache9 commented Jan 18, 2023

Uh oh!

dongjoon-hyun commented Jan 18, 2023

Uh oh!

Apache9 commented Jan 19, 2023

Uh oh!

dongjoon-hyun commented Jan 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Jan 10, 2023•
edited
Loading

ndimiduk commented Jan 10, 2023•
edited
Loading

dongjoon-hyun commented Jan 10, 2023•
edited
Loading