[Python] pyarrow.concat_arrays segfaults if a resulting StringArray's capacity overflows

[asfimport]

I'm sorry if this was already reported, but there's an overflow issue in concatenation of large arrays

In [1]: import pyarrow as pa

In [2]: str_array = pa.array(['a' * 128] * 10**8)

In [3]: large_array = pa.concat_arrays([str_array] * 50)
Segmentation fault (core dumped)

I suppose that this should be handled by upcast to large_string.

Reporter: Artem KOZHEVNIKOV / @artemru

Related issues:

• .take silently overflow on list array (when casting to large_list is needed) (relates to)
• [C++][Parquet] Support LargeListArray (relates to)
• [C++] Take on string chunked arrays slow and fails (is related to)
• [C++][Python] Large strings cause ArrowInvalid: offset overflow while concatenating arrays (is superceded by)

_{Note: This issue was originally created as ARROW-10172. Please see the migration documentation for further details.}

[asfimport]

Artem KOZHEVNIKOV / @artemru:
btw, casting to large_string is not supported neither (it's maybe linked):

str_array.cast(pa.large_string())
ArrowNotImplementedError Traceback (most recent call last)
<ipython-input-2-7399d0bc10ae> in <module>
----> 1 str_array.cast(pa.large_string())

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.ChunkedArray.cast()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/compute.py in cast(arr, target_type, safe)
 85 else:
 86 options = _pc.CastOptions.unsafe(target_type)
---> 87 return call_function("cast", [arr], options)
 88 
 89 

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/_compute.pyx in pyarrow._compute.call_function()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/_compute.pyx in pyarrow._compute.Function.call()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Unsupported cast from string to large_utf8 using function cast_large_string
​

[asfimport]

Artem KOZHEVNIKOV / @artemru:
btw, in pyarrow=2, the behaviour has changed

type(pa.array(['a' * 128] * 10**8))
pyarrow.lib.ChunkedArray # before it was pyarrow.lib.StringArray
type(pa.array(['a' * 128] * 10**8, pa.large_string()))
pyarrow.lib.LargeStringArray 

pa.concat_arrays will overflow

str_array = pa.array(['a' * 128] * 10**8)
pa.concat_arrays(str_array.chunks)
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
<ipython-input-49-ec7499df74ad> in <module>
----> 1 pa.concat_arrays(str_array.chunks)

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.concat_arrays()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/opt/conda/envs/model/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: offset overflow while concatenating arrays

and upcast to large string does not work neither, so it looks like we cannot handle such large arrays correctly as now.

[asfimport]

David Li / @lidavidm:
I happened to see this. The segfault is gone, casting is fixed, but the error is still there.

>>> import pyarrow as pa
>>> str_array = pa.array(['a' * 128] * 10**8)
>>> str_array
<pyarrow.lib.ChunkedArray object at 0x7f7198a89c70>
...
>>> type(str_array)
<class 'pyarrow.lib.ChunkedArray'>
>>> str_array.cast(pa.large_string())
<pyarrow.lib.ChunkedArray object at 0x7f71ca5bd400>
...
>>> large_array = pa.concat_arrays([str_array] * 50)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "pyarrow/array.pxi", line 2884, in pyarrow.lib.concat_arrays
TypeError: Iterable should contain Array objects, got <class 'pyarrow.lib.ChunkedArray'> instead
>>> pa.concat_arrays(str_array.chunks)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "pyarrow/array.pxi", line 2889, in pyarrow.lib.concat_arrays
 File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
 File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays

There's more discussion now in ARROW-17828 so I'm going to close this in favor of that one to keep things in one place