GH-41475: [Python] Build with Python 3.13

[tacaswell]

Rationale for this change

The private function _Py_IsFinalizing renamed to Py_IsFinalizing in Python 3.13.0a1. Without this change pyarrow will not compile with this version of Python or higher.

What changes are included in this PR?

add a version-gated #define to handle the change.

Are these changes tested?

Local build succeeds.

Are there any user-facing changes?

no

• GitHub Issue: [Python] _Py_IsFinalizing will be removed in Python 3.13 #41475

[github-actions]

⚠️ GitHub issue #41475has been automatically assigned in GitHub to PR creator.

[pitrou]

Instead of doing such adjustements one by one, how about we vendor a version of https://github.com/python/pythoncapi-compat ? cc @jorisvandenbossche

[jorisvandenbossche]

Instead of doing such adjustements one by one, how about we vendor a version of https://github.com/python/pythoncapi-compat ?

I am not very familiar with it to estimate how much we would make use of it (this PR also essentially only adds 3 lines, and the actual changes to change _Py_IsFinalizing to Py_IsFinalizing is still something that we would had to do). But yes, if that tool is exactly meant for this, maybe why not just use it.

I quickly tested its upgrade script on the pyarrow C++ code, and what it does change is replace a bunch of .. == Py_None cases with Py_IsNone(..).

But maybe the more interesting usage of the tool would be that we can use newer C API functions in our code that might make our PyArrow C++ easier to read or maintain? (but again not sure how much of those cases would actually apply here)

[jorisvandenbossche]

(I am also fine with considering https://github.com/python/pythoncapi-compat for a follow-up issue, and at least ensure now pyarrow builds with Python 3.13)

[pitrou]

I quickly tested its upgrade script on the pyarrow C++ code, and what it does change is replace a bunch of .. == Py_None cases with Py_IsNone(..).

But maybe the more interesting usage of the tool would be that we can use newer C API functions in our code that might make our PyArrow C++ easier to read or maintain? (but again not sure how much of those cases would actually apply here)

Yes, the idea would be to use newer C API functions as in this PR. Vendoring it should be trivial: there is a single .h file.

[jorisvandenbossche]

I don't have time this or coming week to look into vendoring pythoncapi-compat, ad given the upcoming feature freeze for 17.0 the week after, I would suggest we merge this for now and track using pythoncapi-compat as a follow-up issue.

[kou]

+1

[raulcd]

@github-actions crossbow submit -g python

[raulcd]

I am running extra CI for this. If tests are successful I'll merge as is and we can open a follow up issue to vendor pythoncapi-compat as suggested

[github-actions]

Revision: 79c37cd

Submitted crossbow builds: ursacomputing/crossbow @ actions-d83aa38dd0

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3

[raulcd]

Failures are unrelated

[raulcd]

I've created the follow up issue #43069 to vendor https://github.com/python/pythoncapi-compat

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 1ce69ec.

There were 12 benchmark results with an error:

• Commit Run on ec2-r5-4xlarge-us-east-2 at 2024-06-27 00:44:45Z
  • partitioned-dataset-filter (R) with dataset=dataset-taxi-parquet, language=R, query=payment_type_3
  • partitioned-dataset-filter (R) with dataset=dataset-taxi-parquet, language=R, query=payment_type_3
• and 10 more (see the report linked below)

There was 1 benchmark result indicating a performance regression:

• Commit Run on ec2-m5-4xlarge-us-east-2 at 2024-06-27 00:51:47Z
  • partitioned-dataset-filter (R) with dataset=dataset-taxi-parquet, language=R, query=vignette

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.