[Java] The offset buffer of empty BaseVariableWidthVector should not be empty when exposed through C Data Interface

[viirya]

Describe the bug, including details regarding any error messages, version, and platform.

We encountered an error when exchanging string array from Java to Rust through Arrow C data interface. At Rust side, it complains that the buffer at position 1 (offset buffer) is null. After tracing down and some debugging, it looks like the issue is Java Arrow BaseVariableWidthVector class assigns an empty offset buffer if the array is empty (value count 0).

According to Arrow spec for variable size binary layout:

The offsets buffer contains length + 1 signed integers ...

So for an empty string array, its offset buffer should be a buffer with one element (generally it is 0).

Component(s)

Java

[vibhatha]

@viirya Thanks for reporting this issue. Planning to make a PR?

[viirya]

@vibhatha Yea, I'm working on a fix locally. But it causes a few tests failed now. Still looking into fixing the tests.

[vibhatha]

Wonderful!

[viirya]

It looks like both cases are acceptable (empty or one single zero value element):

https://lists.apache.org/thread/w7g1zfqrjxx0bvrct0mt5zwxvdnc9nob

Close this for now.

[viirya]

Per more discussions in the PR, we probably need to fix C data interface of Java Arrow to properly export empty offset buffer for var-size arrays.

[vibhatha]

Is there a sample code that we could use to reproduce the issue and probably look for a fix?

[viirya]

Hmm, the code producing the issue is complicated. We execute TPCDS query in Spark/DataFusion and pass the results as Arrow vectors through C Data interface to Rust (arrow-rs).

I saw there are some tests in c module RoundtripTest . I'm looking to have a reproducer test there. But I cannot build c module on Mac...