GH-43536: [Python] Do not use borrowed references APIs by lysnikolaou · Pull Request #43540 · apache/arrow

lysnikolaou · 2024-08-02T17:27:41Z

Rationale for this change

For better reference safety under Python free-threaded builds (i.e. with the GIL removed), we should be using Py(List|Dict)_GetItemRef that return strong references and are implemented in a thread-safe manner.

What changes are included in this PR?

Vendor a copy of https://github.com/python/pythoncapi-compat
Port to strong reference APIs for lists and dicts

Are these changes tested?

I ran the tests with the free-threaded build before and after, and there's the same expected failures.

GitHub Issue: [Python] Support for the free-threaded build of CPython 3.13 #43536

github-actions · 2024-08-02T17:28:06Z

⚠️ GitHub issue #43536has been automatically assigned in GitHub to PR creator.

jorisvandenbossche · 2024-08-05T10:19:30Z

General question: are the #ifdef / #else needed, or could we also simply always use the variants that return strong references, also when free-threading is not enabled? Or has that a performance cost?

(or if the answer is that those APIs are not supported for older versions of CPython, adopting pythoncapi-compat might help? #43069)

lysnikolaou · 2024-08-05T11:02:49Z

We could use the strong reference APIs in the non-free-threaded build as well.

Adopting pythoncapi-compat would certainly help. Glad that there's an open issue about that already. I can work on a PR that vendors the compat header.

pitrou · 2024-08-06T13:11:39Z

These changes add a lot of conditional code. Also, sometimes they are not necessary, if we are sure the container (list/dict) will not be mutated in another thread.

lysnikolaou · 2024-08-06T14:59:43Z

Thanks @pitrou for the review! I updated the PR to add pythoncapi_compat.h and use the strong reference APIs from there.

lysnikolaou · 2024-08-13T12:43:47Z

This is ready for another review.

pitrou

LGTM, thanks a lot @lysnikolaou . I will rebase and try to run more CI.

pitrou · 2024-08-15T10:03:40Z

@github-actions crossbow submit -g python -g wheel

github-actions · 2024-08-15T10:07:02Z

Revision: 202de1a

Submitted crossbow builds: ursacomputing/crossbow @ actions-ceaf2f2afb

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-1.26
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-pandas-nightly-numpy-nightly
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0-numpy-1.19
test-conda-python-3.9
test-conda-python-3.9-pandas-latest-numpy-latest
test-conda-python-emscripten
test-cuda-python
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2024-08-15T12:24:25Z

CI failures look unrelated, I'll merge.

conbench-apache-arrow · 2024-08-15T20:36:58Z

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 894f72f.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change #43540 already vendored `pythoncapi_compat.h`, so closing #43069 by using this as well for `Py_IsFinalizing` (which was added in #42034, and for which we opened that follow-up issue to use `pythoncapi_compat.h` instead) Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

### Rationale for this change apache/arrow#43540 already vendored `pythoncapi_compat.h`, so closing apache/arrow#43069 by using this as well for `Py_IsFinalizing` (which was added in apache/arrow#42034, and for which we opened that follow-up issue to use `pythoncapi_compat.h` instead) Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

github-actionsBot added Component: Python awaiting review Awaiting review labels Aug 2, 2024

pitrou requested changes Aug 6, 2024
View reviewed changes

github-actionsBot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 6, 2024

github-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Aug 6, 2024

pitrou reviewed Aug 6, 2024
View reviewed changes

Comment threadpython/pyarrow/src/arrow/python/platform.h Outdated

pitrou reviewed Aug 6, 2024
View reviewed changes

Comment threadpython/pyarrow/src/arrow/python/pythoncapi_compat.h Outdated

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 6, 2024

lysnikolaou mentioned this pull request Aug 7, 2024
[Python] Support for the free-threaded build of CPython 3.13 #43536
Closed
11 tasks

pitrou reviewed Aug 7, 2024
View reviewed changes

Comment threadpython/pyarrow/src/arrow/python/platform.h Outdated

pitrou reviewed Aug 7, 2024
View reviewed changes

Comment threadpython/pyarrow/src/arrow/python/vendored/pythoncapi_compat.h Outdated

pitrou changed the title ~~GH-43536: [Python] Do not use borrowed references APIs under free-threaded CPython~~GH-43536: [Python] Do not use borrowed references APIsAug 15, 2024

pitrou approved these changes Aug 15, 2024
View reviewed changes

lysnikolaou added 5 commits August 15, 2024 12:03

Do not use borrowed references APIs under free-threaded CPython
a427bd9

Add pythoncapi_compat and use strong reference APIs
ba84ac0

Move pythoncapi_compat.h to vendored subdirectory
d713b2a

Fix clang-format issues
dd76029

Do not include pythoncapi_compat.h from public headers
202de1a

pitrou force-pushed the remove-borrowed-references branch from 174a1ef to 202de1aCompare August 15, 2024 10:03

pitrou added 2 commits August 15, 2024 12:07

Fix lint
3f97169

Add RAT exclusion for vendored C++ files in PyArrow
7352f19

pitrou requested review from assignUser, jonkeane, kou and raulcd as code owners August 15, 2024 10:18

pitrou merged commit 894f72f into apache:mainAug 15, 2024

pitrou removed the awaiting change review Awaiting change review label Aug 15, 2024

jorisvandenbossche mentioned this pull request Aug 20, 2024
GH-43069: [Python] Use Py_IsFinalizing from pythoncapi_compat.h #43767
Merged

This was referenced Sep 9, 2024
GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build #43671
Merged
[Python] Fix threading bugs under the free-threaded version related to borrowed references and Pandas initialization #44046
Closed

Uh oh!

Conversation

lysnikolaou commented Aug 2, 2024• edited by pitrou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Uh oh!

github-actionsBot commented Aug 2, 2024

Uh oh!

jorisvandenbossche commented Aug 5, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lysnikolaou commented Aug 5, 2024

Uh oh!

pitrou commented Aug 6, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou commented Aug 6, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou commented Aug 13, 2024

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou commented Aug 15, 2024

Uh oh!

github-actionsBot commented Aug 15, 2024

Uh oh!

pitrou commented Aug 15, 2024

Uh oh!

conbench-apache-arrowBot commented Aug 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lysnikolaou commented Aug 2, 2024•
edited by pitrou
Loading

jorisvandenbossche commented Aug 5, 2024•
edited
Loading