GH-39001: [Java] Modularize remaining modules by jduo · Pull Request #39221 · apache/arrow

jduo · 2023-12-13T23:27:26Z

Rationale for this change

Modularize remaining modules outside of memory modules, vector, and format.

What changes are included in this PR?

Are these changes tested?

Yes, existing unit tests now run with modules when using JDK9+.

Are there any user-facing changes?

Yes. There are new command-line options that may be necessary. The way of specifying the output directory for
JNI native library builds differs. The flight-grpc module has been eliminated since it is now built into flight-core.
Documentation has been updated for these changes.

This PR includes breaking changes to public APIs.
There are a number of package structure changes and some modules now need additional command-line arguments.

Closes: [Java] Build remaining modules as JPMS modules #39001

github-actions · 2023-12-13T23:28:08Z

⚠️ GitHub issue #39001has been automatically assigned in GitHub to PR creator.

jduo · 2023-12-13T23:41:58Z

dataset is having issues due to the directory holding native libs in JARs (the arch eg x86_64) is being treated as a package name. flight-core and other Flight modules requires additional command-line arguments.

Netty is omitted due to requiring user-unfriendly command-line arguments.

jduo · 2024-01-06T13:57:31Z

Doc changes are required as different modules require more command line arguments. I'll update the docs.

assignUser · 2024-01-11T04:06:59Z

FYI for testing runing the java-jars job might be useful. You can start it by commenting @github-action crossbow submit java-jars

jduo · 2024-01-11T21:39:58Z

FYI for testing runing the java-jars job might be useful. You can start it by commenting @github-action crossbow submit java-jars

Thanks, I'll run that now.

jduo · 2024-01-11T21:42:32Z

@GitHub-Action crossbow submit java-jars

jduo · 2024-01-11T22:03:03Z

@github-actions crossbow submit java-jars

github-actions · 2024-01-11T22:05:17Z

Revision: 35f7cbaf1f4efc4742dcc931acd125ad3ff0e8e4

Submitted crossbow builds: ursacomputing/crossbow @ actions-e879fe00cc

Task	Status
java-jars

jduo · 2024-01-11T22:27:03Z

@github-actions crossbow submit java-jars

Use ARROW_JNI_JAVA_<library>_[BIN|LIB]DIR instead of overriding CMAKE_INSTALL_[BIN|LIB]DIR

Remove explicit setting of ARROW_JAVA_JNI_ARCH_DIR in build system. Clarify that this should get auto-detected.

jduo · 2024-01-19T19:47:53Z

@github-actions crossbow submit java-jars

github-actions · 2024-01-19T19:50:27Z

Revision: c5723d1

Submitted crossbow builds: ursacomputing/crossbow @ actions-8e395163f9

Task	Status
java-jars

jduo · 2024-01-19T20:37:04Z

@github-actions crossbow submit java-jars

github-actions · 2024-01-19T20:39:15Z

Revision: 0ece855

Submitted crossbow builds: ursacomputing/crossbow @ actions-0785af577e

Task	Status
java-jars

jduo · 2024-01-19T21:38:39Z

@kou , I've switched to using CMAKE_SYSTEM_PROCESSOR and changed CI scripts and POM files to let auto-detection figure ARROW_JAVA_JNI_ARCH_DIR. I think this addresses everything.

kou

+1 for the CMake part

conbench-apache-arrow · 2024-01-20T02:15:05Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 92682f0.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 10 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change Modularize remaining modules outside of memory modules, vector, and format. ### What changes are included in this PR? ### Are these changes tested? Yes, existing unit tests now run with modules when using JDK9+. ### Are there any user-facing changes? Yes. There are new command-line options that may be necessary. The way of specifying the output directory for JNI native library builds differs. The flight-grpc module has been eliminated since it is now built into flight-core. Documentation has been updated for these changes. **This PR includes breaking changes to public APIs.** There are a number of package structure changes and some modules now need additional command-line arguments. * Closes: apache#39001 Authored-by: James Duong <james.duong@improving.com> Signed-off-by: David Li <li.davidm96@gmail.com>

raulcd · 2024-04-15T09:43:03Z

@github-actions crossbow submit verify-rc-source-integration-linux-*

raulcd · 2024-04-15T09:44:20Z

I am verifying if this caused: #41201

github-actions · 2024-04-15T09:45:22Z

Revision: 0ece855

Submitted crossbow builds: ursacomputing/crossbow @ actions-c29cc1b749

Task	Status
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-20.04-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64

### Rationale for this change Modularize remaining modules outside of memory modules, vector, and format. ### What changes are included in this PR? ### Are these changes tested? Yes, existing unit tests now run with modules when using JDK9+. ### Are there any user-facing changes? Yes. There are new command-line options that may be necessary. The way of specifying the output directory for JNI native library builds differs. The flight-grpc module has been eliminated since it is now built into flight-core. Documentation has been updated for these changes. **This PR includes breaking changes to public APIs.** There are a number of package structure changes and some modules now need additional command-line arguments. * Closes: apache#39001 Authored-by: James Duong <james.duong@improving.com> Signed-off-by: David Li <li.davidm96@gmail.com>

github-actionsBot added the awaiting review Awaiting review label Dec 13, 2023

github-actionsBot added the Component: Java label Dec 13, 2023

kou changed the title ~~GH-39001: Modularize remaining modules~~GH-39001: [Java] Modularize remaining modulesDec 13, 2023

jduo force-pushed the modularize-remaining branch from 9ba832c to 9701a21Compare January 5, 2024 22:55

github-actionsBot added the Component: Documentation label Jan 5, 2024

jduo marked this pull request as ready for review January 6, 2024 00:55

jduo requested review from assignUser, kou, lidavidm and raulcd as code owners January 6, 2024 00:55

jduo force-pushed the modularize-remaining branch 3 times, most recently from ed7b6be to c4c67b1Compare January 10, 2024 01:02

jduo force-pushed the modularize-remaining branch from c4c67b1 to ad83e89Compare January 11, 2024 17:50

github-actionsBot removed the Component: Documentation label Jan 11, 2024

jduo force-pushed the modularize-remaining branch 3 times, most recently from 72747ae to b4d9662Compare January 11, 2024 18:53

github-actionsBot added the Component: Documentation label Jan 11, 2024

jduo added the Breaking Change Includes a breaking change to the API label Jan 11, 2024

jduo self-assigned this Jan 11, 2024

jduo added 4 commits January 19, 2024 11:28

Rename CMAKE_ARCH_DIR to ARROW_JAVA_JNI_ARCH_DIR
d45cbc2

Removed a reference to flight-grpc from docs
184467a

Change build variable name
aaedda7
Use ARROW_JNI_JAVA_<library>_[BIN|LIB]DIR instead of overriding CMAKE_INSTALL_[BIN|LIB]DIR

Derive arch in CMakeLists.txt instead of scripts
3607ed8

jduo force-pushed the modularize-remaining branch from 02c0b82 to 3607ed8Compare January 19, 2024 19:29

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jan 19, 2024

PR suggestions
c5723d1
Remove explicit setting of ARROW_JAVA_JNI_ARCH_DIR in build system. Clarify that this should get auto-detected.

Fix Windows build
0ece855

kou approved these changes Jan 19, 2024
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Jan 19, 2024

lidavidm approved these changes Jan 19, 2024
View reviewed changes

lidavidm merged commit 92682f0 into apache:mainJan 19, 2024

lidavidm removed the awaiting merge Awaiting merge label Jan 19, 2024

github-actionsBot added the awaiting merge Awaiting merge label Jan 19, 2024

alinaliBQ deleted the modularize-remaining branch May 22, 2025 21:21

Uh oh!

Conversation

jduo commented Dec 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Dec 13, 2023

Uh oh!

jduo commented Dec 13, 2023

Uh oh!

jduo commented Jan 6, 2024

Uh oh!

assignUser commented Jan 11, 2024

Uh oh!

jduo commented Jan 11, 2024

Uh oh!

jduo commented Jan 11, 2024

Uh oh!

jduo commented Jan 11, 2024

Uh oh!

github-actionsBot commented Jan 11, 2024

Uh oh!

jduo commented Jan 11, 2024

Uh oh!

jduo commented Jan 19, 2024

Uh oh!

github-actionsBot commented Jan 19, 2024

Uh oh!

jduo commented Jan 19, 2024

Uh oh!

github-actionsBot commented Jan 19, 2024

Uh oh!

jduo commented Jan 19, 2024

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrowBot commented Jan 20, 2024

Uh oh!

raulcd commented Apr 15, 2024

Uh oh!

raulcd commented Apr 15, 2024

Uh oh!

github-actionsBot commented Apr 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jduo commented Dec 13, 2023•
edited
Loading