GH-37484: [Python] Add a FixedSizeTensorScalar class by rok · Pull Request #37533 · apache/arrow

rok · 2023-09-04T04:10:00Z

Rationale for this change

When working with FixedSizeTensorArray we want to access individual tensors. This would be enabled by adding:

defFixedSizeTensorScalar(pa.ExtensionScalar):
defto_numpy_ndarray(): ...

See #37484.

What changes are included in this PR?

This adds FixedSizeTensorScalar and tests for it.

Are there any user-facing changes?

Yes, when calling FixedSizeTensorArray[i] we would get back FixedSizeTensorScalar instead of ExtensionScalar.

Closes: [Python] Add a FixedSizeTensorScalar class #37484

github-actions · 2023-09-04T04:10:30Z

⚠️ GitHub issue #37484has been automatically assigned in GitHub to PR creator.

alippai · 2023-09-04T23:58:38Z

Would the numpy array api or https://data-apis.org/array-api/latest/purpose_and_scope.html add any value here?

rok · 2023-09-05T09:58:43Z

@alippai This PR would effectively implement a __pos__(i) method. Which should per array-api return an array while pyarrow typically returns scalar. I'm not sure how to reconcile this. We should probably have a broader discussion on the mailing list about adopting array API.

AlenkaF

Thanks for working on this! Added two suggestions, otherwise the Pyhton part LGTM.

AlenkaF · 2023-09-19T09:18:57Z

Would the numpy array api or https://data-apis.org/array-api/latest/purpose_and_scope.html add any value here?

I am very much hoping we could implement DLPack in Arrow: #33984. Specially for the new tensor arrays, it would be very beneficial!

@alippai This PR would effectively implement a __pos__(i) method. Which should per array-api return an array while pyarrow typically returns scalar. I'm not sure how to reconcile this. We should probably have a broader discussion on the mailing list about adopting array API.

+1

AlenkaF · 2023-09-19T09:22:41Z

One more thing, can the change in the C++ code (GetTensor feature) also be reflected in the title of the PR and the description? I am not sure how we want to rename it, so I haven't made any change.

jorisvandenbossche

Didn't yet look in detail, but added some quick drive-by comments. And thanks for working on this!

Can you also add some tests for the new Scalar class?

Currently, for the Python bindings, you added a get_tensor(i) method on the array class, but wouldn't make sense to (also/instead) add a to_tensor() method on the scalar class, since this is to get a Tensor for a single element (scalar) of the array?

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

rok

Thanks a lot for the review @pitrou ! Unfortunately I couldn't work on it immediately.
I think I addressed all of your points.

rok · 2024-02-02T13:01:28Z

We discussed this here: #33948 (review)
I would personally prefer to_numpy, but perhaps we can have a discussion about this under a separate issue?
cc @jorisvandenbossche @AlenkaF

rok · 2024-02-02T13:09:22Z

checked_pointer_cast causes a segfault here. Not sure if it's an issue or not.

rok · 2024-02-02T13:13:16Z

As per #37533 (review)ravel shouldn't cause copy if memory layout doesn't change. And we're not actively trying to change the memory layout here.

pitrou

Thanks a lot for the update. This is really getting good, just a couple comments on specific points.

pitrou · 2024-02-07T10:39:15Z

Hmm, I wouldn't be so sure, example:
>>>obj=np.arange(24).reshape(2,3,4)[:,::2] >>>objarray([[[ 0, 1, 2, 3], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [20, 21, 22, 23]]]) >>>obj.strides (96, 64, 8) >>>permutation= (-np.array(obj.strides)).argsort(kind='stable') >>>permutationarray([0, 1, 2]) # permutation is ok>>>values=np.ravel(obj, order="K") >>>valuesarray([ 0, 1, 2, 3, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 22, 23]) >>>values[0] =999>>>valuesarray([999, 1, 2, 3, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 22, 23]) >>>objarray([[[ 0, 1, 2, 3], [ 8, 9, 10, 11]], [[12, 13, 14, 15], [20, 21, 22, 23]]]) # values is a copy! which we can also check using:>>>values.ctypes.data==obj.ctypes.dataFalse# base addresses are different
Of course, we may not really care about this, since the conversion is probably correct anyway. But we may want to add a mention in the docstring that the conversion is only zero-copy if the input array is contiguous.

pitrou · 2024-02-07T10:42:10Z

Not sure if it's an issue either, but it would be worth taking a debugger and understanding exactly what happens :-)

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

rok · 2024-02-08T00:34:49Z

Thanks a lot for the update. This is really getting good, just a couple comments on specific points.

Thanks for the helpful review @pitrou, I'm happy to see this moving forward! I've addressed your points, please let me know if more changes are needed.

pitrou · 2024-02-08T09:43:59Z

as_strided can be a later PR if desired. The docstring addition is good for now!

pitrou · 2024-02-08T09:46:03Z

@github-actions crossbow submit -g python -g wheel

pitrou · 2024-02-08T09:46:48Z

It may be nice to later add a doc section for tensors here:
https://arrow.apache.org/docs/python/

github-actions · 2024-02-08T09:49:59Z

Revision: bf2ca0e

Submitted crossbow builds: ursacomputing/crossbow @ actions-2a16c8cab9

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-38-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

rok · 2024-02-08T11:33:49Z

Added an issue for the docs: #39998
Thanks for your reviews @pitrou @jorisvandenbossche @AlenkaF @alippai !

jorisvandenbossche · 2024-02-08T12:32:09Z

 and the rest of the dimensions will match the permuted shape of the fixed
 shape tensor.

+ The conversion is zero-copy.


Small nit: this is only if the conversion to numpy is zero-copy (i.e. primitive numeric data without nulls)

Good point, added to VariableShapeTensor PR 8ca3bf7

conbench-apache-arrow · 2024-02-08T14:05:17Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 026188e.

There were 9 benchmark results indicating a performance regression:

Commit Run on ursa-i9-9960x at 2024-02-08 11:36:59Z
- tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-02, scale_factor=1
- tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-03, scale_factor=1
and 7 more (see the report linked below)

The full Conbench report has more details. It also includes information about 7 possible false positives for unstable benchmarks that are known to sometimes produce them.

github-actionsBot added Component: Python awaiting committer review Awaiting committer review labels Sep 4, 2023

rok force-pushed the 37484 branch from ccf3696 to bbdc50aCompare September 17, 2023 20:15

github-actionsBot added the Component: C++ label Sep 17, 2023

rok marked this pull request as ready for review September 17, 2023 20:17

rok force-pushed the 37484 branch from bbdc50a to 2ff0210Compare September 18, 2023 12:35

AlenkaF reviewed Sep 19, 2023
View reviewed changes

Comment threadpython/pyarrow/array.pxi Outdated
Comment threadpython/pyarrow/array.pxi Outdated
Comment threadpython/pyarrow/types.pxi Outdated

github-actionsBot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting committer review Awaiting committer review awaiting changes Awaiting changes labels Sep 24, 2023

rok force-pushed the 37484 branch from 151da9d to bbc2fd1Compare October 28, 2023 11:20

AlenkaF added this to the 15.0.0 milestone Nov 16, 2023

jorisvandenbossche reviewed Dec 1, 2023
View reviewed changes

Comment threadcpp/src/arrow/extension/fixed_shape_tensor.cc Outdated
Comment threadcpp/src/arrow/extension/fixed_shape_tensor.cc Outdated
Comment threadpython/pyarrow/array.pxi Outdated

github-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 1, 2023

rok force-pushed the 37484 branch from dc9a805 to 81d4996Compare December 2, 2023 04:35

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 2, 2023

rok force-pushed the 37484 branch from 81d4996 to 80efc48Compare December 2, 2023 04:37

github-actionsBot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Dec 2, 2023

rokand others added 9 commits January 29, 2024 22:56

Review feedback
76874e5

Review feedback
14e2e09

Review feedback
00c5d47

Rename cell->element
6a40b4b

test
a7f9b31

work
da35639

Update cpp/src/arrow/extension/fixed_shape_tensor_test.cc
977a92b
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

review feedback
cc1ba4c

Review feedback
92e3b55

rok commented Feb 2, 2024
View reviewed changes

pitrou reviewed Feb 7, 2024
View reviewed changes

rokand others added 5 commits February 7, 2024 20:25

Apply suggestions from code review
3593031
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lint
bfdeafd

changes to from_numpy_ndarray
83d2934

change to checked casts
e1a1d28

cast to ExtensionArray instead
052eec2

rok mentioned this pull request Feb 8, 2024
[Python] Consider renaming FixedShapeTensorArray.to_numpy_ndarray to FixedShapeTensorArray.to_numpy #39991
Open

pitrou approved these changes Feb 8, 2024
View reviewed changes

Nit: remove extraneous empty line.
bf2ca0e

rok mentioned this pull request Feb 8, 2024
[Docs] Add a doc section for tensor arrays #39998
Open

jorisvandenbossche reviewed Feb 8, 2024
View reviewed changes

rok mentioned this pull request Feb 8, 2024
GH-38007: [C++] Add VariableShapeTensor implementation #38008
Merged

llama90 mentioned this pull request Feb 18, 2024
[Python] Error when executing the command to build pyarrow #40117
Closed

jorisvandenbossche mentioned this pull request Aug 8, 2024
[Python] FixedShapeTensorArray.to_numpy_ndarray() fails with numpy arrays of type string in Pyarrow-16, but works with v15 #43614
Open

Uh oh!

Conversation

rok commented Sep 4, 2023• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Sep 4, 2023

Uh oh!

alippai commented Sep 4, 2023

Uh oh!

rok commented Sep 5, 2023

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlenkaF commented Sep 19, 2023

Uh oh!

AlenkaF commented Sep 19, 2023

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Feb 8, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou commented Feb 8, 2024

Uh oh!

pitrou commented Feb 8, 2024

Uh oh!

github-actionsBot commented Feb 8, 2024

Uh oh!

rok commented Feb 8, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrowBot commented Feb 8, 2024

Uh oh!

Reviewers

Assignees

rok commented Sep 4, 2023•
edited by github-actions Bot
Loading