[Docs] Clarify whether the data buffers of Binary|String View can contain null buffers

[andishgar]

Describe the enhancement requested

As mentioned here, the Arrow specification does not mention here, here, and here whether the data buffers of Binary|String View can be null or not, and this has already become problematic here.

Component(s)

Documentation

[pitrou]

@alamb @tustvold @paleolimbot @CurtHagenlocher What do you think?

[CurtHagenlocher]

I don't have a strong feeling either way, but my bias is to think that -- all other things being equal -- fewer illegal states means that less validation is required and is therefore better. A null buffer doesn't have any more ambiguity than a non-null buffer that's not referenced by any values.

[paleolimbot]

At least for the C export/import, I believe any buffer is allowed to be null if its length would otherwise be zero ( https://arrow.apache.org/docs/format/CDataInterface.html#c.ArrowArray.buffers ), which I think would point to allowing null data buffers in this case.

[pitrou]

At least for the C export/import, I believe any buffer is allowed to be null if its length would otherwise be zero ( https://arrow.apache.org/docs/format/CDataInterface.html#c.ArrowArray.buffers ), which I think would point to allowing null data buffers in this case.

Well, the "empty buffers may be null" rule makes most sense for buffers that are mandatory for the layout. For variadic buffers it is not obvious that null buffers may be useful or desirable.

[paleolimbot]

We could discourage variadic buffers for this reason (I agree that they're probably not useful in this context), but as an importer one would still have to consider that this might happen given that it's been documented as such for a few years.

[alamb]

We recently went through something related to this in arrow-rs (for Utf8/Binary, not Utf8View and BInaryView) with @atwam and polars. Apparently their reading of the spec expects a non null (empty) buffer even when there is are no rows

• arrow-ipc writer does not comply with spec for empty variable-size arrays arrow-rs#9716

We chose to follow the C/C++ implementation (see here apache/arrow-rs#9716 (comment)) which writes non null buffers in this case

Following what @paleolimbot says, it seems like we have existing implementations (and we may not know of them all) that can write null buffers; This mean that in practice, readers will likely have to handle null buffers even if we change the spec to not officially allow it, to be tolerant on read

Therefore I would personally suggest we clarify the spec to say that null buffers are allowed in IPC/C Data Interface and that they are semantically equivalent to a zero length, non null buffer.

[atwam]

More specifically for the issue above (apache/arrow-rs#9716), the problem wasn't just about null vs non-null buffer, but rather the fact that a Utf8/Binary view is supposed to have an offset buffer of length N+1, so with a [0] content for an empty Utf8/Binary array, but some implementations left them to be fully empty or null.

I think it's related, but kind of orthogonal to whether a [] buffer needs to be explicitely encoded or can remain null. Even if an empty buffer can be expressed with a null buffer, in the issue above we should still have a length-1 offset buffer for an empty array (at least as per the specs)

[paleolimbot]

I dug up the last time I thought about this, when it appears that C# and Arrow C++ at one point required non-null data buffers for some other types. I believe both were fixed shortly after I wrote the comment.

https://github.com/apache/arrow-nanoarrow/blob/50ddea7fae57aeb77228637dca32d1e5f0ca4e07/src/nanoarrow/common/array.c#L541-L547

[pitrou]

More specifically for the issue above (apache/arrow-rs#9716), the problem wasn't just about null vs non-null buffer, but rather the fact that a Utf8/Binary view is supposed to have an offset buffer of length N+1

There is no offsets buffer in BinaryView. There is a views buffer which is supposed to be length N (not N+1).