[C++][Python] Casting extension types with list-based storage segfaults

[spenczar]

Describe the bug, including details regarding any error messages, version, and platform.

I'm taking a table and casting it to its own schema. This should always work, but it errors in a certain case.

(this is deep in some library code - please forgive how strange this scenario is; just trust me that it really does happen!)

# An extension type which wraps list-of-int32:
class MyExtensionType(pa.PyExtensionType):
 def __init__(self):
 pa.PyExtensionType.__init__(self, pa.list_(pa.int32()))
 def __reduce__(self):
 return MyExtensionType, (),

# an array of this type:
ext_array = pa.array([[1, 2, 3]], MyExtensionType())
# Wrap that in a struct:
struct_array = pa.StructArray.from_arrays([ext_array], "x")
# Wrap the struct in a table:
table = pa.Table.from_arrays([struct_array], ["field1"])

All of the above is fine, and then:

table.cast(table.schema)

Blows up:

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:3616, in pyarrow.lib.Table.cast()

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:3798, in pyarrow.lib.Table.from_arrays()

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/table.pxi:2962, in pyarrow.lib.Table.validate()

File ~/.pyenv/versions/3.10.10/envs/general-3.10/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Column 0: In chunk 0: Invalid: Struct child array #0 invalid: Invalid: Expected 1 child arrays in array of type list<item: int32>, got 0

I can print table just fine:

pyarrow.Table
field1: struct<x: extension<arrow.py_extension_type<MyExtensionType>>>
 child 0, x: extension<arrow.py_extension_type<MyExtensionType>>
----
field1: [
 -- is_valid: all not null
 -- child 0 type: extension<arrow.py_extension_type<MyExtensionType>>
[[1,2,3]]]

This is pyarrow version 12.0.1.

Component(s)

Python

[spenczar]

This seems to happen as well for FixedShapeTensor tensor types, which indicates it happens for fixed-size list types.

[AlenkaF]

I can reproduce the first issue but cannot reproduce it for FixedShapeTensor array:

>>> import pyarrow as pa
>>> import pyarrow as pa

>>> tensor_type = pa.fixed_shape_tensor(pa.int32(), (2, 2))
>>> arr = [[1, 2, 3, 4], [10, 20, 30, 40], [100, 200, 300, 400]]
>>> storage = pa.array(arr, pa.list_(pa.int32(), 4))
>>> tensor_array = pa.ExtensionArray.from_storage(tensor_type, storage)

>>> data = [
... pa.array([1, 2, 3]),
... pa.array(['foo', 'bar', None]),
... tensor_array,
... ]
>>> my_schema = pa.schema([('f0', pa.int8()),
... ('f1', pa.string()),
... ('tensors_int', tensor_type)])
>>> table = pa.Table.from_arrays(data, schema=my_schema)

>>> table.cast(table.schema)
pyarrow.Table
f0: int8
f1: string
tensors_int: extension<arrow.fixed_shape_tensor>
----
f0: [[1,2,3]]
f1: [["foo","bar",null]]
tensors_int: [[[1,2,3,4],[10,20,30,40],[100,200,300,400]]]

I am not sure if the issue is in the struct or the underlying list type. But it seems to me that the casting of the column in the table.cast is behaving erroneous

arrow/python/pyarrow/table.pxi

Line 3694 in 4fc2731

casted = column.cast(field.type, safe=safe, options=options)

and I think this should work but it segfaults:

>>> struct_array = pa.StructArray.from_arrays([ext_array], "x") 
>>> struct_array.type
StructType(struct<x: extension<arrow.py_extension_type<MyExtensionType>>>)

>>> struct_array.cast(struct_array.type)
/Users/alenkafrim/repos/arrow/cpp/src/arrow/array/array_nested.cc:210: Check failed: (data->child_data.size()) == (1) 
0 libarrow.1300.0.0.dylib 0x0000000142366294 _ZN5arrow4util7CerrLog14PrintBackTraceEv + 52
1 libarrow.1300.0.0.dylib 0x000000014236623c _ZN5arrow4util7CerrLogD2Ev + 120
2 libarrow.1300.0.0.dylib 0x0000000142366184 _ZN5arrow4util7CerrLogD1Ev + 28
3 libarrow.1300.0.0.dylib 0x00000001423661b0 _ZN5arrow4util7CerrLogD0Ev + 28
4 libarrow.1300.0.0.dylib 0x0000000142366078 _ZN5arrow4util8ArrowLogD2Ev + 112
5 libarrow.1300.0.0.dylib 0x00000001423660bc _ZN5arrow4util8ArrowLogD1Ev + 28
6 libarrow.1300.0.0.dylib 0x0000000141d4f5f4 _ZN5arrow8internal11SetListDataINS_8ListTypeEEEvPNS_13BaseListArrayIT_EERKNSt3__110shared_ptrINS_9ArrayDataEEENS_4Type4typeE + 536
7 libarrow.1300.0.0.dylib 0x0000000141d4eee8 _ZN5arrow9ListArray7SetDataERKNSt3__110shared_ptrINS_9ArrayDataEEE + 36
8 libarrow.1300.0.0.dylib 0x0000000141d4ee34 _ZN5arrow9ListArrayC2ENSt3__110shared_ptrINS_9ArrayDataEEE + 68
9 libarrow.1300.0.0.dylib 0x0000000141d4ef58 _ZN5arrow9ListArrayC1ENSt3__110shared_ptrINS_9ArrayDataEEE + 28
10 libarrow.1300.0.0.dylib 0x0000000141ec14bc _ZNSt3__120__shared_ptr_emplaceIN5arrow9ListArrayENS_9allocatorIS2_EEEC2B6v15006IJRKNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 124
11 libarrow.1300.0.0.dylib 0x0000000141ec1430 _ZNSt3__120__shared_ptr_emplaceIN5arrow9ListArrayENS_9allocatorIS2_EEEC1B6v15006IJRKNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 36
12 libarrow.1300.0.0.dylib 0x0000000141ec13b0 _ZNSt3__115allocate_sharedB6v15006IN5arrow9ListArrayENS_9allocatorIS2_EEJRKNS_10shared_ptrINS1_9ArrayDataEEEEvEENS5_IT_EERKT0_DpOT1_ + 60
13 libarrow.1300.0.0.dylib 0x0000000141ec1368 _ZNSt3__111make_sharedB6v15006IN5arrow9ListArrayEJRKNS_10shared_ptrINS1_9ArrayDataEEEEvEENS3_IT_EEDpOT0_ + 52
14 libarrow.1300.0.0.dylib 0x0000000141eb0494 _ZN5arrow12_GLOBAL__N_116ArrayDataWrapper5VisitINS_8ListTypeEEENS_6StatusERKT_ + 52
15 libarrow.1300.0.0.dylib 0x0000000141e97ccc _ZN5arrow15VisitTypeInlineINS_12_GLOBAL__N_116ArrayDataWrapperEJEEENS_6StatusERKNS_8DataTypeEPT_DpOT0_ + 1164
16 libarrow.1300.0.0.dylib 0x0000000141e974d8 _ZN5arrow9MakeArrayERKNSt3__110shared_ptrINS_9ArrayDataEEE + 124
17 libarrow.1300.0.0.dylib 0x000000014209c47c _ZN5arrow14ExtensionArray7SetDataERKNSt3__110shared_ptrINS_9ArrayDataEEE + 284
18 libarrow.1300.0.0.dylib 0x000000014209c320 _ZN5arrow14ExtensionArrayC2ERKNSt3__110shared_ptrINS_9ArrayDataEEE + 84
19 libarrow.1300.0.0.dylib 0x000000014209c51c _ZN5arrow14ExtensionArrayC1ERKNSt3__110shared_ptrINS_9ArrayDataEEE + 36
20 libarrow_python.dylib 0x00000001066d2914 _ZNSt3__120__shared_ptr_emplaceIN5arrow14ExtensionArrayENS_9allocatorIS2_EEEC2B6v15006IJRNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 100
21 libarrow_python.dylib 0x00000001066d25f0 _ZNSt3__120__shared_ptr_emplaceIN5arrow14ExtensionArrayENS_9allocatorIS2_EEEC1B6v15006IJRNS_10shared_ptrINS1_9ArrayDataEEEEEES4_DpOT_ + 36
22 libarrow_python.dylib 0x00000001066d24f8 _ZNSt3__115allocate_sharedB6v15006IN5arrow14ExtensionArrayENS_9allocatorIS2_EEJRNS_10shared_ptrINS1_9ArrayDataEEEEvEENS5_IT_EERKT0_DpOT1_ + 60
23 libarrow_python.dylib 0x00000001066d1240 _ZNSt3__111make_sharedB6v15006IN5arrow14ExtensionArrayEJRNS_10shared_ptrINS1_9ArrayDataEEEEvEENS3_IT_EEDpOT0_ + 52
24 libarrow_python.dylib 0x00000001066d1160 _ZNK5arrow2py15PyExtensionType9MakeArrayENSt3__110shared_ptrINS_9ArrayDataEEE + 224
25 libarrow.1300.0.0.dylib 0x0000000141eb0814 _ZN5arrow12_GLOBAL__N_116ArrayDataWrapper5VisitERKNS_13ExtensionTypeE + 84
26 libarrow.1300.0.0.dylib 0x0000000141e97e10 _ZN5arrow15VisitTypeInlineINS_12_GLOBAL__N_116ArrayDataWrapperEJEEENS_6StatusERKNS_8DataTypeEPT_DpOT0_ + 1488
27 libarrow.1300.0.0.dylib 0x0000000141e974d8 _ZN5arrow9MakeArrayERKNSt3__110shared_ptrINS_9ArrayDataEEE + 124
28 libarrow.1300.0.0.dylib 0x0000000141d578d8 _ZNK5arrow11StructArray5fieldEi + 368
29 libarrow.1300.0.0.dylib 0x00000001420b907c _ZN5arrow12_GLOBAL__N_112ArrayPrinter5VisitERKNS_11StructArrayE + 468
30 libarrow.1300.0.0.dylib 0x00000001420b3aa0 _ZN5arrow16VisitArrayInlineINS_12_GLOBAL__N_112ArrayPrinterEJEEENS_6StatusERKNS_5ArrayEPT_DpOT0_ + 1308
31 libarrow.1300.0.0.dylib 0x00000001420b19d0 _ZN5arrow12_GLOBAL__N_112ArrayPrinter5PrintERKNS_5ArrayE + 60
32 libarrow.1300.0.0.dylib 0x00000001420b1b6c _ZN5arrow11PrettyPrintERKNS_5ArrayERKNS_18PrettyPrintOptionsEPNSt3__113basic_ostreamIcNS6_11char_traitsIcEEEE + 68
33 libarrow.1300.0.0.dylib 0x00000001420b1bbc _ZN5arrow11PrettyPrintERKNS_5ArrayERKNS_18PrettyPrintOptionsEPNSt3__112basic_stringIcNS6_11char_traitsIcEENS6_9allocatorIcEEEE + 68
34 lib.cpython-310-darwin.so 0x0000000105490a6c _ZL41__pyx_pf_7pyarrow_3lib_5Array_33to_stringP29__pyx_obj_7pyarrow_3lib_Arrayiiiib + 488
35 lib.cpython-310-darwin.so 0x0000000105481a68 _ZL41__pyx_pw_7pyarrow_3lib_5Array_34to_stringP7_objectS0_S0_ + 1628
36 Python 0x0000000104787cfc cfunction_call + 60
37 lib.cpython-310-darwin.so 0x00000001053a1e8c _ZL19__Pyx_PyObject_CallP7_objectS0_S0_ + 140
38 lib.cpython-310-darwin.so 0x00000001053a7280 _ZL24__Pyx_PyObject_CallNoArgP7_object + 120
39 lib.cpython-310-darwin.so 0x000000010547e380 _ZL39__pyx_pf_7pyarrow_3lib_5Array_37__str__P29__pyx_obj_7pyarrow_3lib_Array + 324
40 lib.cpython-310-darwin.so 0x000000010547ce30 _ZL39__pyx_pw_7pyarrow_3lib_5Array_38__str__P7_object + 28
41 Python 0x000000010478cfac PyObject_Str + 172
42 Python 0x00000001047b97cc unicode_new + 116
43 Python 0x00000001047a4ad8 type_call + 84
44 lib.cpython-310-darwin.so 0x00000001053a1e8c _ZL19__Pyx_PyObject_CallP7_objectS0_S0_ + 140
45 lib.cpython-310-darwin.so 0x0000000105653be0 _ZL26__Pyx__PyObject_CallOneArgP7_objectS0_ + 176
46 lib.cpython-310-darwin.so 0x00000001053a2b24 _ZL25__Pyx_PyObject_CallOneArgP7_objectS0_ + 116
47 lib.cpython-310-darwin.so 0x000000010547d6b8 _ZL40__pyx_pf_7pyarrow_3lib_5Array_31__repr__P29__pyx_obj_7pyarrow_3lib_Array + 552
48 lib.cpython-310-darwin.so 0x000000010547ce00 _ZL40__pyx_pw_7pyarrow_3lib_5Array_32__repr__P7_object + 28
49 Python 0x000000010478d17c PyObject_Repr + 156
50 Python 0x000000010474fe78 PyFile_WriteObject + 392
51 Python 0x000000010488d7c4 sys_displayhook + 200
52 Python 0x00000001047886fc cfunction_vectorcall_O + 100
53 Python 0x00000001048277d8 _PyEval_EvalFrameDefault + 15056
54 Python 0x0000000104822ab4 _PyEval_Vector + 376
55 Python 0x0000000104822928 PyEval_EvalCode + 104
56 Python 0x000000010487e998 run_eval_code_obj + 84
57 Python 0x000000010487e8fc run_mod + 112
58 Python 0x000000010487ec9c PyRun_InteractiveOneObjectEx + 656
59 Python 0x000000010487dbcc _PyRun_InteractiveLoopObject + 124
60 Python 0x000000010487d46c _PyRun_AnyFileObject + 76
61 Python 0x00000001048819a4 PyRun_AnyFileExFlags + 84
62 Python 0x00000001048a04a0 pymain_run_stdin + 160
63 Python 0x000000010489fa98 Py_RunMain + 512
64 Python 0x00000001048a1000 Py_BytesMain + 40
65 dyld 0x0000000191d6ff28 start + 2236
zsh: abort python

[spenczar]

Thanks for taking a look @AlenkaF. Here is a reproducer for FixedShapeTensor - it only happens if you go through the struct. The issue is mis-titled. It's actually a bug with Tables which contain a Struct field which have an ExtensionType which has a ListTyped-storage (whew, that's a lot!):

In [38]: import pyarrow as pa

In [39]: tensor_type = pa.fixed_shape_tensor(pa.int32(), (2, 2))

In [40]: arr = [[1, 2, 3, 4], [10, 20, 30, 40], [100, 200, 300, 400]]

In [41]: storage = pa.array(arr, pa.list_(pa.int32(), 4))

In [42]: tensor_array = pa.ExtensionArray.from_storage(tensor_type, storage)
 ...:

In [43]: pa.StructArray.from_arrays([tensor_array], "x")
Out[43]:
<pyarrow.lib.StructArray object at 0x154e5a680>
-- is_valid: all not null
-- child 0 type: extension<arrow.fixed_shape_tensor>
 [
 [
 1,
 2,
 3,
 4
 ],
 [
 10,
 20,
 30,
 40
 ],
 [
 100,
 200,
 300,
 400
 ]
 ]

In [44]: struct_array = pa.StructArray.from_arrays([tensor_array], "x")

In [45]: table = pa.Table.from_arrays([struct_array], ["field1"])

In [46]: table.cast(table.schema)
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[46], line 1
----> 1 table.cast(table.schema)

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/table.pxi:3616, in pyarrow.lib.Table.cast()

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/table.pxi:3798, in pyarrow.lib.Table.from_arrays()

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/table.pxi:2962, in pyarrow.lib.Table.validate()

File ~/.pyenv/versions/3.10.12/envs/adam-core/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Column 0: In chunk 0: Invalid: Struct child array #0 invalid: Invalid: Expected 1 child arrays in array of type fixed_size_list<item: int32>[4], got 0

[NellyWhads]

Is this related to #34985?

[AlenkaF]

I asked Copilot to go through the backtrace and I checked the result. It does seem that, in both cases, when casting an extension type with list-based storage, the cast kernel produces ArrayData with the correct type but drops the child data. When the extension type then tries to wrap this into an ExtensionArray and calls MakeArray, it tries to construct a list array from that incomplete ArrayData and crashes.

In case that helps anybody starting to look for a fix. Will also update the title to reflect the underlying bug.

[connorsimms]

Hello! I've looked into this issue and I've diagnosed the cause in array/data.cc. I've got a fix with C++/Python tests, and i've also checked it against the Python snippets in this thread. If no one is working on this, I'd be more than happy to open a PR :).

[connorsimms]

take