GH-38997: [Java] Modularize format and vector by jduo · Pull Request #38995 · apache/arrow

jduo · 2023-11-30T00:08:36Z

Rationale for this change

This PR depends on #39011 .

Make arrow-vector and arrow-format JPMS modules for improved compatibility with newer JDKs
and tools that require modules.

What changes are included in this PR?

add module-info.java files for vector and format.

Are these changes tested?

Yes, these modules are run as JPMS modules in JDK9+ unit tests.

Are there any user-facing changes?

Yes, some test classes have moved and users can now run these as JPMS modules.

Closes: [Java] Build arrow-format and arrow-vector as JPMS modules #38997

github-actions · 2023-11-30T00:09:01Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-11-30T02:11:35Z

⚠️ GitHub issue #38997has been automatically assigned in GitHub to PR creator.

lidavidm · 2023-12-04T21:05:09Z

This is on my backlog - I'm hoping to give this a review this week

jduo · 2023-12-05T20:02:24Z

Thanks for the review @lidavidm . This shouldn't go in yet until we're ready to bite the bullet on modularizing memory-core + making the command-line change though (or we figure out how to avoid the command-line change). I'm changing this back to draft for the time being.

lidavidm · 2024-01-09T21:01:34Z

@davisusanibar can you help take a look at the nullness annotation issues here?

jduo · 2024-01-10T01:04:15Z

@davisusanibar can you help take a look at the nullness annotation issues here?

I fixed a null annotation issue with CountingAllocationListener in the parent patch (#39011).

lidavidm · 2024-01-10T01:28:07Z

Ah whoops, I thought I was looking at that one.

lidavidm · 2024-01-11T14:20:52Z

I think this just needs to be rebased?

Do not add tests in org.apache.arrow.util because that is an exported package for arrow-memory-core and causes module conflicts

Having an implicit dependency from arrow-memory-core does not put immutables on the module-path when running tests and causes module issues.

Export public package for arrow-vector module. Allow jackson to use arrow-vector pojo classes reflectively for serialization.

Needed because Vector classes use MemoryUtil.UNSAFE.

Also don't depend on jdk.unsupported

Needed because it uses MemoryUtil.UNSAFE

jduo · 2024-01-11T17:52:19Z

I think this just needs to be rebased?

Yes, this is rebased now @lidavidm

conbench-apache-arrow · 2024-01-13T07:04:55Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 1a622ec.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change This PR depends on apache#39011 . Make arrow-vector and arrow-format JPMS modules for improved compatibility with newer JDKs and tools that require modules. ### What changes are included in this PR? * add module-info.java files for vector and format. ### Are these changes tested? Yes, these modules are run as JPMS modules in JDK9+ unit tests. ### Are there any user-facing changes? Yes, some test classes have moved and users can now run these as JPMS modules. * Closes: apache#38997

github-actionsBot added Component: Java awaiting review Awaiting review labels Nov 30, 2023

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch from 4947db9 to 54446ecCompare November 30, 2023 00:28

jduo changed the title ~~Modularize format and vector and add plugin for compiling module-info files with JDK8 builds~~GH-38997: [Java] Modularize format and vector and add plugin for compiling module-info files with JDK8 buildsNov 30, 2023

jduo marked this pull request as ready for review November 30, 2023 03:12

jduo requested a review from lidavidm as a code owner November 30, 2023 03:12

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch from 0b49cfa to 83bab66Compare December 1, 2023 12:36

lidavidm approved these changes Dec 5, 2023
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Dec 5, 2023

jduo marked this pull request as draft December 5, 2023 20:02

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch from 83bab66 to 0342cb3Compare December 7, 2023 23:41

jduo marked this pull request as ready for review December 8, 2023 00:23

jduo changed the title ~~GH-38997: [Java] Modularize format and vector and add plugin for compiling module-info files with JDK8 builds~~GH-38997: [Java] Modularize format and vectorDec 8, 2023

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch 3 times, most recently from 47030a0 to f39d07bCompare December 11, 2023 21:02

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch 4 times, most recently from b99f597 to ccd5ffcCompare January 5, 2024 21:51

jduo requested review from assignUser, kou and raulcd as code owners January 5, 2024 21:51

github-actionsBot added the Component: Documentation label Jan 5, 2024

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch from ccd5ffc to 038549aCompare January 5, 2024 22:24

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch 2 times, most recently from cedbd00 to ae7bacbCompare January 9, 2024 19:22

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch from ae7bacb to 1e7d5e9Compare January 10, 2024 01:01

jduo added 11 commits January 11, 2024 09:45

Add module-info.java files for format and vector
733a775

Move arrow-vector tests to use a different package
b523285
Do not add tests in org.apache.arrow.util because that is an exported package for arrow-memory-core and causes module conflicts

Explicitly have arrow-vector depend on immutables
367f590
Having an implicit dependency from arrow-memory-core does not put immutables on the module-path when running tests and causes module issues.

Export packages for arrow-vector module
0476125
Export public package for arrow-vector module. Allow jackson to use arrow-vector pojo classes reflectively for serialization.

Fix a missed reference to moved arrow-vector test classes
558a041

Have arrow-vector depend on jdk.unsupported
b7977f5
Needed because Vector classes use MemoryUtil.UNSAFE.

Remove unused dependencies on netty and java.sql in arrow-vector
d2a0839

Add dependency on eclipse collections to arrow-vector
f9ad830

Have arrow-vector depend on memory-core
1a8e084
Also don't depend on jdk.unsupported

Have arrow-vector rely on Unsafe
f35501e
Needed because it uses MemoryUtil.UNSAFE

Use correct JPMS name for SLF4J in arrow-vector
94b9c20

jduo force-pushed the modularize-format-and-vector-and-add-plugin branch from 1e7d5e9 to 94b9c20Compare January 11, 2024 17:47

github-actionsBot removed the Component: Documentation label Jan 11, 2024

lidavidm approved these changes Jan 11, 2024
View reviewed changes

jduo merged commit 1a622ec into apache:mainJan 11, 2024

alinaliBQ deleted the modularize-format-and-vector-and-add-plugin branch May 22, 2025 21:21

Navigation Menu

Uh oh!

GH-38997: [Java] Modularize format and vector - #38995

GH-38997: [Java] Modularize format and vector#38995
jduo merged 11 commits into
apache:mainfrom
Bit-Quill:modularize-format-and-vector-and-add-plugin

jduo commented Nov 30, 2023•
edited
Loading

Uh oh!

github-actionsBot commented Nov 30, 2023

Uh oh!

github-actionsBot commented Nov 30, 2023

Uh oh!

lidavidm commented Dec 4, 2023

Uh oh!

jduo commented Dec 5, 2023

Uh oh!

lidavidm commented Jan 9, 2024

Uh oh!

jduo commented Jan 10, 2024

Uh oh!

lidavidm commented Jan 10, 2024

Uh oh!

lidavidm commented Jan 11, 2024

Uh oh!

jduo commented Jan 11, 2024

Uh oh!

conbench-apache-arrowBot commented Jan 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jduo commented Nov 30, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Nov 30, 2023

Uh oh!

github-actionsBot commented Nov 30, 2023

Uh oh!

lidavidm commented Dec 4, 2023

Uh oh!

jduo commented Dec 5, 2023

Uh oh!

lidavidm commented Jan 9, 2024

Uh oh!

jduo commented Jan 10, 2024

Uh oh!

lidavidm commented Jan 10, 2024

Uh oh!

lidavidm commented Jan 11, 2024

Uh oh!

jduo commented Jan 11, 2024

Uh oh!

conbench-apache-arrowBot commented Jan 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jduo commented Nov 30, 2023•
edited
Loading