GH-343: Fix ListVector offset buffer not properly serialized for nested empty arrays by Yicong-Huang · Pull Request #967 · apache/arrow-java

Yicong-Huang · 2026-01-17T00:18:07Z

What's Changed

Fix ListVector/LargeListVector IPC serialization when valueCount is 0.

Problem

When valueCount == 0, setReaderAndWriterIndex() was setting offsetBuffer.writerIndex(0), which means readableBytes() == 0. IPC serializer uses readableBytes() to determine buffer size, so 0 bytes were written to the IPC stream. This crashes IPC readers in other libraries because Arrow spec requires offset buffer to have at least one entry [0].

@viirya:

The offset buffers are allocated properly. But during IPC serialization, they are ignored.
 public long readableBytes() {
return writerIndex - readerIndex;
}
So when ListVector.setReaderAndWriterIndex() sets writerIndex(0) and readerIndex(0), readableBytes() returns 0 - 0 = 0.
Then when MessageSerializer.writeBatchBuffers() calls WriteChannel.write(buffer), it writes 0 bytes.
So the flow is:
valueCount=0 → ListVector.setReaderAndWriterIndex() sets offsetBuffer.writerIndex(0)
VectorUnloader.getFieldBuffers() returns the buffer with writerIndex=0
MessageSerializer.writeBatchBuffers() writes the buffer
WriteChannel.write(buffer) checks buffer.readableBytes() which is 0
0 bytes are written to the IPC stream
PyArrow read the batch with the missing buffer → crash when other libraries to read

Fix

Simplify setReaderAndWriterIndex() to always use (valueCount + 1) * OFFSET_WIDTH for offset buffer's writerIndex. When valueCount == 0, this correctly sets writerIndex to OFFSET_WIDTH, ensuring offset[0] is included in serialization.

Testing

Added tests for nested empty lists verifying offset buffer has correct readableBytes().

Closes#343.

jbonofre · 2026-01-19T13:41:50Z

@Yicong-Huang can you please rebase the PR ? Thanks !

viirya · 2026-01-21T03:47:14Z

When outer array is empty, nested writers are never invoked, so child list's offset buffer remains unallocated (capacity = 0). This violates Arrow spec which requires offset[0] = 0.

I think you are referring the writers in Spark. It is out of context here and not related to the root cause. We should update the description to explain the issue clearly.

The offset buffers are actually allocated properly. But during IPC serialization, they are ignored.

publiclongreadableBytes() {
returnwriterIndex - readerIndex;
}

So when ListVector.setReaderAndWriterIndex() sets writerIndex(0) and readerIndex(0), readableBytes() returns 0 - 0 = 0.

Then when MessageSerializer.writeBatchBuffers() calls WriteChannel.write(buffer), it writes 0 bytes.

So the flow is:

valueCount=0 → ListVector.setReaderAndWriterIndex() sets offsetBuffer.writerIndex(0)
VectorUnloader.getFieldBuffers() returns the buffer with writerIndex=0
MessageSerializer.writeBatchBuffers() writes the buffer
WriteChannel.write(buffer) checks buffer.readableBytes() which is 0
0 bytes are written to the IPC stream
PyArrow read the batch with the missing buffer → crash when other libraries to read

viirya · 2026-01-21T03:54:42Z

Hi @lidavidm @jbonofre, do you think this can catch up the Arrow Java 19.0.0 release?

viirya · 2026-01-21T20:16:03Z

- validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
- offsetBuffer.writerIndex((valueCount + 1) * OFFSET_WIDTH);
- }
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));


When valueCount == 0, I think validity buffer writer index should be validityBuffer.writerIndex(0);.

I believe when valueCount==0, BitVectorHelper.getValidityBufferSizeFromCount(valueCount) also returns 0, so it is equivalent. The current version might be simpler. If you prefer an if branch to handle it separately, I can also apply it.

I added the if branch back to handle valueCount==0 case. But I still think it is not necessary?

Oh, if BitVectorHelper.getValidityBufferSizeFromCount(valueCount) returns 0 for valueCount == 0, then it is okay.

viirya · 2026-01-21T22:15:13Z

+ // Allocate outer only - simulates case where inner is never written to
+ outerList.allocateNew();
+ outerList.setValueCount(0);


I think innerList should also call allocateNew? Not allocated is different to not written.

viirya · 2026-01-21T22:15:27Z

+ // Only allocate level0 - simulates case where all nested levels are empty
+ level0.allocateNew();
+ level0.setValueCount(0);


viirya · 2026-01-21T23:37:15Z

+ try (LargeListVector outerList = LargeListVector.empty("outer", allocator)) {
+ // Setup LargeList<LargeList<Int>>
+ outerList.addOrGetVector(FieldType.nullable(MinorType.LARGELIST.getType()));
+ LargeListVector innerList = (LargeListVector) outerList.getDataVector();
+ innerList.addOrGetVector(FieldType.nullable(MinorType.INT.getType()));


I looked these tests again. I think this nested structure is not necessary for the unit tests here. It only matter for our usage at Spark side (on the ArrowWriters). But for here, we just need to make sure that a ListVector/LargeListVector has meaningful and correct readableBytes value after they are allocated.
Maybe we can simplify these tests.

viirya

I only have some comments on the tests to simplify them. Otherwise, the fix looks correct to me.

@jbonofre @lidavidm Can you take a look and see if we can get this into the next release?

Thank you!

jbonofre · 2026-01-22T17:57:27Z

I'm doing a new review pass.

jbonofre

LGTM, thanks !

viirya · 2026-01-23T17:24:53Z

Thanks you @jbonofre!

Can we backport this fix to versions like 18.3.0 and release 18.3.1?

jbonofre · 2026-01-23T17:37:17Z

@viirya my plan is more to release 19.0.0. Do you really need 18.3.1 ?

viirya · 2026-01-23T18:47:59Z

@viirya my plan is more to release 19.0.0. Do you really need 18.3.1 ?

Because we are using 18.3.0 currently, it will be much safer to upgrade with 18.3.1. Is it possible to backport it to 18.3 and release 18.3.1?

jbonofre · 2026-01-24T05:45:20Z

@viirya ok, I understand. Let me complete 19.0.0 first, I will prepare 18.3.1 after.

Yicong-Huang · 2026-01-24T06:16:53Z

@viirya ok, I understand. Let me complete 19.0.0 first, I will prepare 18.3.1 after.

Thanks a lot @viirya @jbonofre!

viirya · 2026-01-24T06:17:53Z

@viirya ok, I understand. Let me complete 19.0.0 first, I will prepare 18.3.1 after.

Thank you so much! @jbonofre

viirya

@Yicong-Huang Should we also fix setReaderAndWriterIndex at BaseVariableWidthVector?

Yicong-Huang · 2026-01-26T23:33:09Z

@Yicong-Huang Should we also fix setReaderAndWriterIndex at BaseVariableWidthVector?

sure will also fix the base!

Yicong-Huang · 2026-02-10T19:07:16Z

Hi @jbonofre, just wanted to check if there’s any rough ETA for the 19.0.0 or 18.3.1 release. This would help us plan our dependency upgrade. Really appreciate all the work on the releases. Thanks!

cc @viirya

Yicong-Huang · 2026-03-09T16:07:02Z

Hi @jbonofre, just wanted to check if there’s any rough ETA for the 19.0.0 or 18.3.1 release. This would help us plan our dependency upgrade. Really appreciate all the work on the releases. Thanks!
cc @viirya

Just gently following up on this when you have a chance. Thanks again.

…offset buffer serialization (#989) ## What's Changed Fix `BaseVariableWidthVector`/`BaseLargeVariableWidthVector` IPC serialization when `valueCount` is 0. ### Problem When `valueCount == 0`, `setReaderAndWriterIndex()` was setting `offsetBuffer.writerIndex(0)`, which means `readableBytes() == 0`. IPC serializer uses `readableBytes()` to determine buffer size, so 0 bytes were written to the IPC stream. This crashes IPC readers in other libraries because Arrow spec requires offset buffer to have at least one entry `[0]`. This is a follow-up to #967 which fixed the same issue in `ListVector`/`LargeListVector`. ### Fix Modify `setReaderAndWriterIndex()` to always use `(valueCount + 1) * OFFSET_WIDTH` for the offset buffer's `writerIndex`, moved outside the if/else branch. When the offset buffer capacity is insufficient (e.g., empty buffer from constructor or loaded via `loadFieldBuffers()`), it reallocates a properly sized buffer on demand. ### Testing Added tests for empty `VarCharVector` and `LargeVarCharVector` verifying offset buffer has correct `readableBytes()` after `setValueCount(0)`. Closes#343 --------- Co-authored-by: Yicong Huang <yicong.huang+data@databricks.com>

viirya · 2026-04-01T22:05:12Z

@viirya ok, I understand. Let me complete 19.0.0 first, I will prepare 18.3.1 after.
Thank you so much! @jbonofre

Hello @jbonofre, thank you for reviewing and preparing 19.0.0 release. Now as 19.0.0 was released, can you help us prepare 18.3.1 release including these fixes if you have time? Thank you!

dmitry-chirkov-dremio · 2026-06-18T04:43:55Z

We are troubleshooting a regression that ends up in following stack:

(java.lang.IndexOutOfBoundsException) readerIndex: 0, writerIndex: 4 (expected: 0 <= readerIndex <= writerIndex <= capacity(0))
io.netty.buffer.AbstractByteBuf.checkIndexBounds():112
io.netty.buffer.AbstractByteBuf.writerIndex():135
io.netty.buffer.NettyArrowBuf.writerIndex():215
io.netty.buffer.NettyArrowBuf.unwrapBuffer():631

Our starting point is this.

Is it possible that merged changes break Netty's assumptions in unwrapBuffer?

edit: looks like change in #989 was more involved (and more correct?)

…lized for nested empty arrays (apache#967)" This reverts commit 0f8a080.

Yicong-Huang requested review from jbonofre, laurentgo, lidavidm and wgtmac as code owners January 17, 2026 00:18

Yicong-Huang changed the title ~~GH-343 Fix ListVector offset buffer not allocated for nested empty arrays~~GH-343: Fix ListVector offset buffer not allocated for nested empty arraysJan 17, 2026

This comment has been minimized.
Sign in to view

lidavidm added the bug-fix PRs that fix a big. label Jan 18, 2026

github-actionsBot added this to the 19.0.0 milestone Jan 18, 2026

fix: offset buffer with empty array
7dbdcc4

Yicong-Huang force-pushed the fix/343-empty-nested-list-offset-buffer branch from 8b09237 to 7dbdcc4Compare January 20, 2026 18:21

fix: resotre allocation size
878c8d5