[C++][Python] `_export_to_c` segmentation fault for `binary_view` array

[drain99]

Describe the bug, including details regarding any error messages, version, and platform.

_export_to_c seems to cause segmentation fault for binary_view array constructed as follows:
ChunkedArray(binary...) --combine--> BinaryArray --cast--> BinaryViewArray

Python version: 3.11.13
PyArrow version: 23.0.1

Reproducer below

import ctypes
import pyarrow as pa

# C Data Interface layouts
class ArrowSchema(ctypes.Structure):
 _fields_ = [
 ("format", ctypes.c_char_p),
 ("name", ctypes.c_char_p),
 ("metadata", ctypes.c_char_p),
 ("flags", ctypes.c_int64),
 ("n_children", ctypes.c_int64),
 ("children", ctypes.POINTER(ctypes.c_void_p)),
 ("dictionary", ctypes.c_void_p),
 ("release", ctypes.c_void_p),
 ("private_data", ctypes.c_void_p),
 ]

class ArrowArray(ctypes.Structure):
 _fields_ = [
 ("length", ctypes.c_int64),
 ("null_count", ctypes.c_int64),
 ("offset", ctypes.c_int64),
 ("n_buffers", ctypes.c_int64),
 ("n_children", ctypes.c_int64),
 ("buffers", ctypes.POINTER(ctypes.c_void_p)),
 ("children", ctypes.POINTER(ctypes.c_void_p)),
 ("dictionary", ctypes.c_void_p),
 ("release", ctypes.c_void_p),
 ("private_data", ctypes.c_void_p),
 ]

c_array = ArrowArray()
c_schema = ArrowSchema()

arrs = [
 # exporting this is fine
 pa.array([b'a', b'e', b'a', b'e'], type=pa.binary_view()),
 # this also works fine
 pa.chunked_array([
 pa.array([b'a', b'e'], type=pa.binary_view()),
 pa.array([b'a', b'e'], type=pa.binary_view())
 ]).combine_chunks(),
 # chunked(binary...) -> binary -> binary_view segfaults
 pa.chunked_array([
 pa.array([b'a', b'e'], type=pa.binary()),
 pa.array([b'a', b'e'], type=pa.binary())
 ]).combine_chunks().cast(pa.binary_view()),
]
print(arrs)
print([arr.validate(full=True) for arr in arrs])
arrs[0]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
arrs[1]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)
# this fails
arrs[2]._export_to_c(ctypes.addressof(c_array),ctypes.addressof(c_schema),)

Segmentation fault (core dumped)

Component(s)

Python

[AlenkaF]

Thank you for opening the issue!
I can reproduce the segfault locally, the repr provided is very clear. I did a bit of looking into the issue and asked Copilot for some help. What I found was that the segfault only happens for short strings and works for long strings, for example:

In [2]: long = pa.chunked_array([
 ...: pa.array([b'a' * 13, b'e' * 13], type=pa.binary()),
 ...: pa.array([b'a' * 13, b'e' * 13], type=pa.binary()),
 ...: ]).combine_chunks().cast(pa.binary_view())

In [3]: long._export_to_c(ctypes.addressof(c_array), ctypes.addressof(c_schema))
 ...: print("Long strings: OK")
 ...: 
Long strings: OK

In [4]: short = pa.chunked_array([
 ...: pa.array([b'a', b'e'], type=pa.binary()),
 ...: pa.array([b'a', b'e'], type=pa.binary()),
 ...: ]).combine_chunks().cast(pa.binary_view())

In [5]: short._export_to_c(ctypes.addressof(c_array), ctypes.addressof(c_schema))
 ...: print("Short strings: OK") # never reached
[1] 18460 segmentation fault ipython

Copilot is pointing out that the bug is in the scalar_cast_string.cc implementation where the extra data buffer is dropped because all_entries_are_inline is True

arrow/cpp/src/arrow/compute/kernels/scalar_cast_string.cc

Lines 465 to 467 in e8b7b4e

	if (all_entries_are_inline) {
	output->buffers[2] = nullptr;
	}

So this would be a bug in the cast kernel. We might also update the bridge file

arrow/cpp/src/arrow/c/bridge.cc

Lines 606 to 614 in e8b7b4e

	if (need_variadic_buffer_sizes) {
	auto variadic_buffers = std::span(data->buffers).subspan(2);
	export_.variadic_buffer_sizes_.resize(variadic_buffers.size());
	size_t i = 0;
	for (const auto& buf : variadic_buffers) {
	export_.variadic_buffer_sizes_[i++] = buf->size();
	}
	export_.buffers_.back() = export_.variadic_buffer_sizes_.data();
	}

guarding against null-pointer variadic buffers.

cc @pitrou

[pitrou]

@AlenkaF Thanks for the analysis. I agree it's probably not ok to have null variadic buffers, we should instead resize the buffers in the cast kernel's output.

We should also update our array validation routines to ensure that null variadic buffers don't pass validation (i.e., in the snippet above, short.validate() should fail but it currently succeeds).

[pitrou]

@andishgar @fenfeng9 @Periecle Is one of you interested in working on this? While the reproducer is in Python, the bug is most probably in the C++ codebase.

[fenfeng9]

@pitrou Sure, I'd be happy to take this one.

[fenfeng9]

A smaller reproduction case:
Casting short binary values to binary_view produces an array whose third buffer slot is None ([None, 32, None]).

And validate(full=True) still succeeds, but _export_to_c segfaults when exporting that null variadic buffer slot.

Reproduce

import ctypes
import platform
import pyarrow as pa


if __name__ == "__main__":
 print(f"Python {platform.python_version()}, PyArrow {pa.__version__}", flush=True)
 print(flush=True)

 arr = pa.array([b"a", b"e"], type=pa.binary())
 print("binary array:", flush=True)
 print(arr, flush=True)
 print(
 f"binary buffers : {[None if buf is None else buf.size for buf in arr.buffers()]}",
 flush=True,
 )

 arr = arr.cast(pa.binary_view())
 print()
 print("after cast to binary_view:", flush=True)
 print(arr, flush=True)
 print(
 f"binary_view buffers : {[None if buf is None else buf.size for buf in arr.buffers()]}",
 flush=True,
 )

 arr.validate(full=True)
 print("validate(full=True) : OK", flush=True)

 print("_export_to_c : calling", flush=True)
 out = ctypes.create_string_buffer(1024)
 arr._export_to_c(ctypes.addressof(out))
 print("_export_to_c : OK", flush=True)

Result

Python 3.12.13, PyArrow 24.0.0

binary array:
[
 61,
 65
]
binary buffers : [None, 12, 2]

after cast to binary_view:
[
 61,
 65
]
binary_view buffers : [None, 32, None]
validate(full=True) : OK
_export_to_c : calling

[andishgar]

@pitrou

This is the minimum C++ code to produce the error:

TEST(CdataInterface, BinaryView) {
 arrow::BinaryViewBuilder builder;
 ASSERT_OK(builder.Append("1"));
 ASSERT_OK_AND_ASSIGN(auto array_0, builder.Finish());
 array_0->data()->buffers.push_back(nullptr);
 struct ArrowArray array_out;
 struct ArrowSchema schema_out;
 ASSERT_OK(arrow::ExportArray(*array_0, &array_out, &schema_out));
}

The root of the problem is the line below, which assumes all data buffers of BinaryView are non‑null:

arrow/cpp/src/arrow/c/bridge.cc

Line 611 in e8b7b4e

export_.variadic_buffer_sizes_[i++] = buf->size();

I have a question: as far as I have read the specification, it does not mention whether the data buffers of BinaryViewType can be null or not. Is there any clarification for that in the Arrow specification?

[andishgar]

@fenfeng9, I'm aware that you submitted a PR. I don't intend to interfere it - I'm only asking to clarify the memory layout of BinaryView.