PHOENIX-6475 Use org.openlabtesting.protobuf:protoc when building on Linux ARM64 by martin-g · Pull Request #1238 · apache/phoenix

martin-g · 2021-05-25T13:26:27Z

Stock Protobuf-Java adds aarch64 protoc binary in 3.5.0.
To be able to build on Linux ARM64 with Protobuf 2.x we should use https://github.com/os72/protoc-jar:2.6.1+

martin-g · 2021-05-25T13:29:39Z

- <protobuf-java.version>2.5.0</protobuf-java.version>
+ <!-- com.google repo will be used except on Aarch64 platform. -->
+ <protobuf.group>com.google.protobuf</protobuf.group>
+ <protobuf-java.version>2.6.1</protobuf-java.version>


I've updated to 2.6.1 to keep the x86_64 and aarch64 versions in sync.

Reverted to 2.5.0 for both x86_64 and aarch64!
The OpenLabTesting guys released 2.5.0.2 which brings statically linked Protoc.
Here is its build script: https://gist.github.com/liusheng/64aee1b27de037f8b9ccf1873b82c413

stoty

This patch changes too much, and we specifically want to use the exact same version (2.5.0) that HBase and Hadoop does.

Please use https://repo1.maven.org/maven2/org/openlabtesting/protobuf/protoc/2.5.0/ for the aarch64 protoc binary,
and only override only the groupId used by the protobuf maven plugin from the profile.

stoty · 2021-05-25T16:18:52Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 20s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 0s	Maven dependency ordering for branch
+1 💚	mvninstall	10m 50s	master passed
+0	hbaserecompile	22m 11s	HBase recompiled.
+1 💚	compile	1m 30s	master passed
+1 💚	javadoc	2m 31s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 19s	Maven dependency ordering for patch
+1 💚	mvninstall	8m 11s	the patch passed
+0	hbaserecompile	14m 51s	HBase recompiled.
+1 💚	compile	1m 32s	the patch passed
+1 💚	javac	1m 32s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 4s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 31s	the patch passed
		_ Other Tests _
-1 ❌	unit	114m 47s	root in the patch failed.
-1 ❌	asflicense	2m 9s	The patch generated 614 ASF License warnings.
		171m 45s

Reason	Tests
Failed junit tests	phoenix.end2end.PhoenixTTLToolIT
	phoenix.end2end.IndexBuildTimestampIT
	phoenix.end2end.IndexVerificationOldDesignIT
	phoenix.end2end.IndexRepairRegionScannerIT
	phoenix.end2end.IndexExtendedIT
	phoenix.end2end.index.GlobalIndexCheckerIT
	phoenix.end2end.ConcurrentMutationsExtendedIT
	phoenix.end2end.IndexToolForPartialBuildIT
	phoenix.end2end.IndexRebuildTaskIT
	phoenix.end2end.index.IndexMetadataIT
	phoenix.end2end.index.ViewIndexIT
	phoenix.end2end.IndexToolForPartialBuildWithNamespaceEnabledIT
	phoenix.end2end.CsvBulkLoadToolIT
	phoenix.end2end.IndexToolTimeRangeIT
	phoenix.end2end.LogicalTableNameExtendedIT
	phoenix.end2end.LogicalTableNameIT
	phoenix.schema.stats.NamespaceDisabledStatsCollectorIT
	phoenix.end2end.ConcurrentUpsertsWithoutIndexedColsIT
	phoenix.schema.stats.NamespaceEnabledStatsCollectorIT
	phoenix.end2end.BackwardCompatibilityIT
	phoenix.end2end.IndexToolIT
	phoenix.end2end.IndexScrutinyToolIT
	phoenix.end2end.IndexToolForNonTxGlobalIndexIT
	phoenix.end2end.IndexToolForDeleteBeforeRebuildIT
	phoenix.end2end.IndexScrutinyToolForTenantIT
	phoenix.end2end.RegexBulkLoadToolIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1238
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 23641dc7f3f2 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `3231aad`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/1/artifact/yetus-general-check/output/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/1/testReport/
asflicense	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/1/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count	11223 (vs. ulimit of 30000)
modules	C: phoenix-core phoenix-pherf . U: .
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/1/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

martin-g · 2021-05-26T05:57:32Z

Please use https://repo1.maven.org/maven2/org/openlabtesting/protobuf/protoc/2.5.0/ for the aarch64 protoc binary,

I've tried it before choosing os72 but it didn't work because:

${user.home}/.m2/repository/org/openlabtesting/protobuf/protoc/${protobuf-java.version}/protoc-${protobuf-java.version}-${os.detected.classifier}.exe has no executable bit

Caused by: java.io.IOException: Cannot run program "/home/ubuntu/.m2/repository/org/openlabtesting/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe": error=13, Permission denied
at java.lang.ProcessBuilder.start (ProcessBuilder.java:1048)
at java.lang.Runtime.exec (Runtime.java:621)
at org.codehaus.plexus.util.cli.Commandline.execute (Commandline.java:660)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLineAsCallable (CommandLineUtils.java:136)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine (CommandLineUtils.java:106)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine (CommandLineUtils.java:89)
at org.xolstice.maven.plugin.protobuf.Protoc.execute (Protoc.java:242)
...

After "chmod +x ..." it started failing with:

[INFO] --- protobuf-maven-plugin:0.6.1:compile (compile-protoc) @ phoenix-core ---
[INFO] Compiling 10 proto file(s) to /home/ubuntu/git/apache/phoenix/phoenix-core/target/generated-sources/protobuf/java
[ERROR] PROTOC FAILED: /home/ubuntu/.m2/repository/org/openlabtesting/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
[ERROR] /home/ubuntu/git/apache/phoenix/phoenix-core/src/main/protobuf/PGuidePosts.proto [0:0]: /home/ubuntu/.m2/repository/org/openlabtesting/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
[ERROR] /home/ubuntu/git/apache/phoenix/phoenix-core/src/main/protobuf/ServerCacheFactory.proto [0:0]: /home/ubuntu/.m2/repository/org/openlabtesting/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe: error while loading shared libraries: libprotoc.so.8: cannot open shared object file: No such file or directory
...

i.e. the executable is dynamically linked

file /home/ubuntu/.m2/repository/org/openlabtesting/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe
/home/ubuntu/.m2/repository/org/openlabtesting/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=ad7c0941f0121611ef1963de1bf6d699e2517c5f, with debug_info, not stripped

and nothing provides libprotoc.so.8 in the LD_LIBRARY_PATH.

I will contact the OpenLab Testing guys and see whether they can release 2.5.0.1 with statically linked executable!

stoty · 2021-05-26T06:13:18Z

Thanks for the explanation and your work on it.

I was suggesting the Openlabtesting version because HBase use(d) that one, and it had 2.5.0 available for arm.

However, if that one has dependency problems, then of course it is not useful for us.

I can see two solutions:

You can abandon this change, and specify the compiled ARM protoc binary location in a system property on the maven command line when building.
Use the (only ?) known working os72 2.6.1 version of protoc in the aarch64 profile.
However, in this case please do not change the x86_64 protobuf artifact for 2.5.0, nor the java library versions, only the protoc artifact groupId and version, and only for the arm profile.
(You'll still need to extract the version and groupId into properties of course)

If you choose 2.) please also add a comment to the profile that we're using 2.6.x here because there is no public 2.5.0 artifact for arm available.

…Linux ARM64 Stock Protobuf-Java adds aarch64 protoc binary in 3.5.0. To be able to build on Linux ARM64 with Protobuf 2.x we should use https://github.com/theopenlab/protobuf

stoty

+1 I lack the hardware to test this, but LGTM

martin-g · 2021-05-26T12:50:28Z

@stoty about hardware to test it: few other Hadoop related projects use Linux ARM64 Jenkins agents. See https://ci-hadoop.apache.org/computer/ and look for arm.
Are you interested to test Phoenix on these too ?

stoty · 2021-05-26T14:10:21Z

In theory it'd be nice to test on ARM, but it's not enough of a priority at the moment to put in the work to set up the tests and maintain them.

martin-g · 2021-05-26T14:18:29Z

I could offer my help doing the extra work and maintenance!
Since Phoenix is mainly Java based I don't expect much work related to ARM64. But having CI to notify us when there is a regression would be nice!

stoty · 2021-05-26T15:05:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 9s	Maven dependency ordering for branch
+1 💚	mvninstall	11m 8s	master passed
+0	hbaserecompile	23m 28s	HBase recompiled.
+1 💚	compile	1m 33s	master passed
+1 💚	javadoc	2m 12s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s	Maven dependency ordering for patch
+1 💚	mvninstall	8m 3s	the patch passed
+0	hbaserecompile	14m 29s	HBase recompiled.
+1 💚	compile	1m 29s	the patch passed
+1 💚	javac	1m 29s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 3s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 13s	the patch passed
		_ Other Tests _
-1 ❌	unit	118m 31s	root in the patch failed.
-1 ❌	asflicense	2m 18s	The patch generated 614 ASF License warnings.
		170m 51s

Reason	Tests
Failed junit tests	phoenix.end2end.AuditLoggingIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1238
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux bbf39ec11ce4 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `3231aad`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/2/artifact/yetus-general-check/output/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/2/testReport/
asflicense	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count	11587 (vs. ulimit of 30000)
modules	C: phoenix-core . U: .
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/2/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

stoty · 2021-05-26T18:28:33Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	7m 3s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+0 🆗	mvndep	5m 1s	Maven dependency ordering for branch
+1 💚	mvninstall	13m 11s	master passed
+0	hbaserecompile	25m 20s	HBase recompiled.
+1 💚	compile	1m 44s	master passed
+1 💚	javadoc	2m 31s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s	Maven dependency ordering for patch
+1 💚	mvninstall	10m 16s	the patch passed
+0	hbaserecompile	17m 45s	HBase recompiled.
+1 💚	compile	1m 37s	the patch passed
+1 💚	javac	1m 37s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	xml	0m 2s	The patch has no ill-formed XML file.
+1 💚	javadoc	2m 44s	the patch passed
		_ Other Tests _
-1 ❌	unit	137m 16s	root in the patch failed.
-1 ❌	asflicense	1m 40s	The patch generated 614 ASF License warnings.
		201m 1s

Reason	Tests
Failed junit tests	phoenix.end2end.BackwardCompatibilityIT
	phoenix.end2end.AuditLoggingIT
	phoenix.end2end.UpsertSelectIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1238
Optional Tests	dupname asflicense javac javadoc unit xml compile
uname	Linux 36b2dbc588e9 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `3231aad`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/3/artifact/yetus-general-check/output/patch-unit-root.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/3/testReport/
asflicense	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count	7635 (vs. ulimit of 30000)
modules	C: phoenix-core . U: .
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1238/3/console
versions	git=2.7.4 maven=3.3.9
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

martin-g commented May 25, 2021
View reviewed changes

Comment threadphoenix-pherf/pom.xml

martin-g commented May 25, 2021
View reviewed changes

stoty suggested changes May 25, 2021
View reviewed changes

Comment threadphoenix-pherf/pom.xml

PHOENIX-6475Use org.openlabtesting.protobuf:protoc when building on …
2938ad6
…Linux ARM64 Stock Protobuf-Java adds aarch64 protoc binary in 3.5.0. To be able to build on Linux ARM64 with Protobuf 2.x we should use https://github.com/theopenlab/protobuf

stoty approved these changes May 26, 2021
View reviewed changes

martin-g changed the title ~~PHOENIX-6475 Use github.com/os72/protoc-jar when building on Linux ARM64~~PHOENIX-6475 Use org.openlabtesting.protobuf:protoc when building on Linux ARM64May 26, 2021

martin-g mentioned this pull request May 26, 2021
OMID-210 Use org.openlabtesting.protobuf:protoc:2.5.0 on Linux ARM64 apache/phoenix-omid#96
Closed

stoty closed this May 27, 2021

Navigation Menu

Uh oh!

PHOENIX-6475 Use org.openlabtesting.protobuf:protoc when building on Linux ARM64 - #1238

PHOENIX-6475 Use org.openlabtesting.protobuf:protoc when building on Linux ARM64#1238
martin-g wants to merge 1 commit into
apache:masterfrom
martin-g:fix-build-on-arm64

martin-g commented May 25, 2021

Uh oh!

Uh oh!

martin-gMay 25, 2021

Uh oh!

martin-gMay 26, 2021

Uh oh!

stoty left a comment

Uh oh!

Uh oh!

stoty commented May 25, 2021

Uh oh!

martin-g commented May 26, 2021•
edited
Loading

Uh oh!

stoty commented May 26, 2021•
edited
Loading

Uh oh!

stoty left a comment

Uh oh!

martin-g commented May 26, 2021•
edited
Loading

Uh oh!

stoty commented May 26, 2021

Uh oh!

martin-g commented May 26, 2021

Uh oh!

stoty commented May 26, 2021

Uh oh!

stoty commented May 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

martin-g commented May 25, 2021

Uh oh!

Uh oh!

martin-gMay 25, 2021

Choose a reason for hiding this comment

Uh oh!

martin-gMay 26, 2021

Choose a reason for hiding this comment

Uh oh!

stoty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stoty commented May 25, 2021

Uh oh!

martin-g commented May 26, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stoty commented May 26, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stoty left a comment

Choose a reason for hiding this comment

Uh oh!

martin-g commented May 26, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stoty commented May 26, 2021

Uh oh!

martin-g commented May 26, 2021

Uh oh!

stoty commented May 26, 2021

Uh oh!

stoty commented May 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martin-g commented May 26, 2021•
edited
Loading

stoty commented May 26, 2021•
edited
Loading

martin-g commented May 26, 2021•
edited
Loading