[Java] Java implementation of Arrow C data interface

[asfimport]

As mentioned in ARROW-7272 or in PR #10201, it would be valuable to have the Arrow C data interface implemented in Java to provide better inter-process data sharing facility without depending on 3rd party serialization protocols.

Reporter: Hongze Zhang / @zhztheplayer
Assignee: Roee Shlomo / @roee88

Related issues:

• [Java] Add support for sliced arrays in C Data Interface (causes)
• [Doc] Update implementation status (causes)
• [Java/Python] in-process vector sharing from Java to Python (relates to)
• [Java] Integration tests for the C data Interface implementation for Java (relates to)
• [Python] Rewrite pyarrow.jvm using the C data interface (relates to)
• [Java] Implement ArrowArrayStream/C Stream Interface (is related to)

PRs and other links:

• GitHub Pull Request apache/arrow#11067

_{Note: This issue was originally created as ARROW-12965. Please see the migration documentation for further details.}

[asfimport]

Roee Shlomo / @roee88:
FYI we have a use case for interop between arrow Java and Rust. I started to implement the C data interface in https://github.com/roee88/arrow/tree/java-ffi/java (ffi module).

[asfimport]

Antoine Pitrou / @pitrou:
@roee88 Are you still interested to work on this? It would be nice to have the C data interface supported by the Arrow Java implementation at some point.

[asfimport]

Roee Shlomo / @roee88:

 Are you still interested to work on this?

Yes I plan to contribute it. Nothing changed in the last two days :)

However, from a testing perspective it seems like the best I can do are some roundtrip tests (like in arrow-rs). A separate effort for C Data Interface integration tests is needed (not just for java) to fully test such implementations. Are there any plans or a design/discussion on it? 

[asfimport]

Antoine Pitrou / @pitrou:
Woops, sorry, for some reason I thought your comment was much older than that :-)

Yes, roundtrip tests are a good start. I agree that integration tests would be desirable; there hasn't been any discussion about them for now.

[asfimport]

Antoine Pitrou / @pitrou:
Note we have Python-based tests for an ad-hoc Python-Java bridge, which may give ideas for how to test the C data interface:
https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_jvm.py

[asfimport]

Roee Shlomo / @roee88:
We noticed that for UnionArray the cpp implementation includes a validity buffer although it shouldn't be there according to the specification. In the arrow2 rust implementation a workaround is used (in the FFI code for Union an empty validity buffer is explicitly added). To me it sounds like a bug that should be resolved in arrow-cpp instead. @pitrou Should we try to workaround it in Java too or open a bug ticket?

[asfimport]

Antoine Pitrou / @pitrou:
Hmm... I think the validity buffer is null, even though the pointer is there. Isn't that the case?

[asfimport]

Roee Shlomo / @roee88:
According to the specification it shouldn't be there at all. https://arrow.apache.org/docs/format/Columnar.html#buffer-listing-for-each-layout

[asfimport]

Antoine Pitrou / @pitrou:
Oh, you're right. Can you open a Jira then?

[asfimport]

Roee Shlomo / @roee88:
ARROW-14179 opened

[asfimport]

Kouhei Sutou / @kou:
Issue resolved by pull request 11067
#11067