[Python] Consider renaming FixedShapeTensorArray.to_numpy_ndarray to FixedShapeTensorArray.to_numpy

[rok]

Describe the enhancement requested

FixedShapeTensorArray.to_numpy_ndarray was introduced in #33948 (comment) however we might want to rename it to FixedShapeTensorArray.to_numpy as proposed here #37533 (comment). This would result in overriding current FixedShapeTensorArray.to_numpy.

Component(s)

Python

[jorisvandenbossche]

I am personally in favor of keeping the current name, for the reasons I mentioned in the linked thread.

The to_numpy() method is an existing method on all pyarrow Arrays, and from which you expect to get a 1D numpy array of the same length as the pyarrow array.

I certainly agree that the 2D array is more useful for a FixedSizeTensorArray, but changing to_numpy() to a 2D array just for this class would break consistency with all other types.
(e.g. generally I expect pa.array(arr.to_numpy()) to work (not necessarily preserving the exact type because of limitations of numpy notwithstanding), but that would no longer be the case when we change the return value for FixedSizeTensorArray)

And if you need to ndarray, we have the alternative method available (we could eg update the docstring of to_numpy to more prominently point to that)

[jorisvandenbossche]

array([array([1, 2, 3, 4, 5, 6], dtype=int8),
 array([ 7, 8, 9, 10, 11, 12], dtype=int8)], dtype=object)

In which circumstances is this a desirable behavior? It even makes a copy, right?

I agree it will typically not be the case, but it can still be the desirable behaviour for any context where you need a 1D array (and expect to_numpy() to give you that). Another example of such context is constructors like pd.Series that expect a 1D array.

FWIW it does not make a copy. Similarly as our ListArray -> numpy convrsion, each subarray is a slice from one converted parent ndarray (of course it still needs to allocate the object-dtype 1D array that hold those zero-copy subarrays).

[pitrou]

I agree it will typically not be the case, but it can still be the desirable behaviour for any context where you need a 1D array (and expect to_numpy() to give you that).

It's a 1D array of 1D arrays. What good is that for exactly?

Edit:

Another example of such context is constructors like pd.Series that expect a 1D array.

Ok. But a Series of Numpy arrays is quite specific and unusual, isn't it?

[jorisvandenbossche]

Ah, I expect it to be a 1D array of ndarrays (to preserve the shape of each individual element). But right now it gives you 1D subarrays I assume because it just uses the storage array's to_numpy.

ListArray and FixedSizeListArray do exactly the same, returning a 1D array of arrays.

But a Series of Numpy arrays is quite specific and unusual, isn't it?

It's not that unusual because that's the only way to store that kind of data in pandas (but that's of course more due pandas' limited data type system)

[rok]

Can we reach some agreement on this?

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

[thisisnic]

This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.

[rok]

Working on #40354 I remembered we haven't reached a consensus on FixedShapeTensorArray.to_numpy vs FixedShapeTensorArray.to_numpy_ndarray. With VariableShapeTensorArray it's less of a dilemma since VariableShapeTensorArray.to_numpy wouldn't make any sense. I'd go with VariableShapeTensorScalar.to_numpy to return a N-dimensional array though.