[Python] Convert multidimensional arrays into FixedShapeTensorArrays automatically in `pyarrow.array`

[alanhdu]

Describe the enhancement requested

Now that there is a canonical fixed_shape_tensor type, I was wondering whether there'd be any appetite for making pyarrow.array automatically convert numpy arrays with >1 dimension into FixedShapeTensorArrays.

You can do this manually by using pa.FixedShapeTensorArray.from_numpy_ndarray, but it seems more convenient for pyarrow.array to automatically support this.

If there's appetite for this, I think I'd be able to contribute a patch.

Component(s)

Python

[AlenkaF]

I am personally interested in this idea 👍
Could you elaborate a bit more how you plan to tackle this feature/which part of the codebase would you update, before making a PR?

[alanhdu]

Sure! I haven't fully groked the codebase, but I think the easiest thing would be to directly update

arrow/python/pyarrow/array.pxi

Line 119 in 3e4eaa9

def array(object obj, type=None, mask=None, size=None, from_pandas=None,

to call into FixedShapeTensorArray.from_numpy_ndarray when necessary. I think that basically means inserting another branch at

arrow/python/pyarrow/array.pxi

Line 323 in 3e4eaa9

result = _ndarray_to_array(values, mask, type, c_from_pandas, safe,

to check if values.ndim > 1.

Al alternative would be to try and update

arrow/python/pyarrow/src/arrow/python/numpy_to_arrow.cc

Line 312 in 3e4eaa9

Status NumPyConverter::Convert() {

to return an ExtensionArray with the FixedShapeTensor if ndim > 1, but it's not clear to me how to go from the CExtensionArray to the Python type without modifying the Cython wrapper (and if we do thta, it's not clear there are advantages to not just doing it all in the Cython wrapper).

[mariosasko]

A bit out of the scope of this issue, I assume, but maybe the best (most generic) solution eventually would be allowing extension types to extend the type inference mechanism so that they can be inferred from ("non-basic") Python objects. This should also address the scenario when these objects are nested, e.g., inside a dictionary/struct.

[AlenkaF]

to return an ExtensionArray with the FixedShapeTensor if ndim > 1, but it's not clear to me how to go from the CExtensionArray to the Python type without modifying the Cython wrapper (and if we do thta, it's not clear there are advantages to not just doing it all in the Cython wrapper).

Yeah, I think if doing any kind of change in the array constructor that would fit in the Python C++ part of the codebase, because that is where all of the logic is implemented.

That said, I agree with @mariosasko that it would actually make sense to do it in a more generic way - to allow extension types the type inference when converting from numpy ndarrays for example.

The changes we are talking about here are not trivial.
@jorisvandenbossche what do you think?

[thisisnic]

This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.