[Python] Support converting list of multi-dimensional arrays to FixedShapeTensor

[jorisvandenbossche]

Describe the enhancement requested

When you want to construct a fixed shape tensor array, not from a single contiguous ndarray, but from a list of the individual array elements, this "accidentally" works for 1d arrays (because we support that for ListArray):

>>> pa.array([np.array([1,2,3]), np.array([2,3,4])], type=pa.fixed_shape_tensor(pa.int64(), shape=(3,)))
<pyarrow.lib.FixedShapeTensorArray object at 0x7fadc4d1f3a0>
[
 [
 1,
 2,
 3
 ],
 [
 2,
 3,
 4
 ]
]

But trying the same for >1 d arrays, you get the following error:

>>> pa.array([np.array([[1,2,3]]), np.array([[2,3,4]])], type=pa.fixed_shape_tensor(pa.int64(), shape=(1, 3)))
...
ArrowInvalid: Can only convert 1-dimensional array values

AFAIK, the only way to currently do this conversion, is to convert the list of arrays to a single ndarray yourself, and then use pa.FixedShapeTensorArray.from_numpy_ndarray.

(cc @rok)

Component(s)

Python

[jorisvandenbossche]

And probably worth a separate issue, but also for the conversion the other way around to python, it would be nice to preserve the shape of the ndarrays:

>>> arr = pa.FixedShapeTensorArray.from_numpy_ndarray(np.stack([np.array([[1,2,3], [3,4,5]]), np.array([[2,3,4], [5,6,7]])], axis=1))
# results in an array of 1D arrays
>>> arr.to_numpy(zero_copy_only=False)
array([array([1, 2, 3, 2, 3, 4]), array([3, 4, 5, 5, 6, 7])], dtype=object)
# the scalars do have a method to convert to the correct shape
>>> [v.to_numpy() for v in arr]
[array([[1, 2, 3],
 [2, 3, 4]]),
 array([[3, 4, 5],
 [5, 6, 7]])]

[rok]

Agreed on both. Let's try to get this into the next release.