GH-37880: [CI][Python][Packaging] Add support for Python 3.12 by raulcd · Pull Request #37901 · apache/arrow

raulcd · 2023-09-27T11:28:14Z

Rationale for this change

Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release.

What changes are included in this PR?

This PR adds jobs to build pyarrow wheels for Python 3.12.

Are these changes tested?

They will be tested via archery tasks

Are there any user-facing changes?

No but users will be able to use pyarrow with Python 3.12

Closes: [Python][Packaging] Support Python 3.12 and upload wheels to PyPI #37880

raulcd · 2023-09-27T11:28:58Z

@github-actions crossbow submit cp312

github-actions · 2023-09-27T11:32:02Z

Revision: 907f66d

Submitted crossbow builds: ursacomputing/crossbow @ actions-6d09e69209

Task	Status
wheel-macos-big-sur-cp312-arm64
wheel-macos-mojave-cp312-amd64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

raulcd · 2023-09-27T12:59:10Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T13:01:46Z

Revision: 5cee4a8

Submitted crossbow builds: ursacomputing/crossbow @ actions-2decf3cc0d

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T13:24:32Z

@github-actions crossbow submit wheel-manylinux-2-28-cp312-amd64

github-actions · 2023-09-27T13:27:09Z

Revision: 54d5d34

Submitted crossbow builds: ursacomputing/crossbow @ actions-ca5ce95faa

Task	Status
wheel-manylinux-2-28-cp312-amd64

raulcd · 2023-09-27T13:32:40Z

The macOS issue seems to fail due to this (https://github.com/ursacomputing/crossbow/actions/runs/6326301003/job/17179575146#step:13:173):
pypa/setuptools#3661
I'll investigate further, here the error log:

 File "/Users/runner/work/crossbow/crossbow/test-amd64-env/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 994, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-bw0gv97b/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
import distutils.core
ModuleNotFoundError: No module named 'distutils'

raulcd · 2023-09-27T13:38:09Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T13:40:44Z

Revision: 349f362

Submitted crossbow builds: ursacomputing/crossbow @ actions-f607b07df1

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T14:50:17Z

I am facing some issues when updating setuptools with numpy:

 File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-2rp89sp0/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
import setuptools.version
File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-2rp89sp0/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
import pkg_resources
File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-2rp89sp0/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
register_finder(pkgutil.ImpImporter, find_on_path)
^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
[end of output]

This seems related to: pypa/pip#12179

raulcd · 2023-09-27T14:50:42Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T14:53:28Z

Revision: 3e3046c

Submitted crossbow builds: ursacomputing/crossbow @ actions-635103f8ad

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T15:26:15Z

@github-actions crossbow submit wheel-manylinux-2-28-cp312-amd64

github-actions · 2023-09-27T15:29:43Z

Revision: 125ac34

Submitted crossbow builds: ursacomputing/crossbow @ actions-70da4f8be9

Task	Status
wheel-manylinux-2-28-cp312-amd64

raulcd · 2023-09-27T16:27:51Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T16:30:51Z

Revision: dbd90cc

Submitted crossbow builds: ursacomputing/crossbow @ actions-93131c2a12

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-27T17:54:43Z

There seems to be an issue compiling grpcio with Python 3.12:
grpc/grpc#33063
It seems it will be released soon as v1.59.0-pre1 seems to contain a fix for it: https://github.com/grpc/grpc/releases/tag/v1.59.0-pre1
I am facing the same issue on the macos wheels here: https://github.com/ursacomputing/crossbow/actions/runs/6328784518/job/17187734569#step:13:10233
and the manylinux one here: https://github.com/ursacomputing/crossbow/actions/runs/6328121689/job/17185543580#step:10:12731
I might disable google testbench temporarily to run the rest of the tests.

raulcd · 2023-09-27T17:59:26Z

@github-actions crossbow submit wheel-macos-mojave-cp312-amd64

github-actions · 2023-09-27T18:02:09Z

Revision: e92c96b

Submitted crossbow builds: ursacomputing/crossbow @ actions-b8b091adfb

Task	Status
wheel-macos-mojave-cp312-amd64

raulcd · 2023-09-28T08:30:32Z

@github-actions crossbow submit cp312

github-actions · 2023-09-28T08:33:10Z

Revision: e92c96b

Submitted crossbow builds: ursacomputing/crossbow @ actions-fa45089ccb

Task	Status
wheel-macos-big-sur-cp312-arm64
wheel-macos-mojave-cp312-amd64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

raulcd · 2023-10-03T07:37:36Z

@github-actions crossbow submit cp312

github-actions · 2023-10-03T07:40:11Z

Revision: b1d98ea5109ace814c0baa62cce326ea9c6e0e7b

Submitted crossbow builds: ursacomputing/crossbow @ actions-817b67d67c

Task	Status
wheel-macos-big-sur-cp312-arm64
wheel-macos-mojave-cp312-amd64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

raulcd · 2023-10-03T10:34:42Z

@github-actions crossbow submit wheel-macos-big-sur-cp312-arm64

raulcd · 2023-10-10T17:17:46Z

@github-actions crossbow submit wheel-macos-mojave-cp310-amd64

github-actions · 2023-10-10T17:20:20Z

Revision: 27a979c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1822e92828

Task	Status
wheel-macos-mojave-cp310-amd64

raulcd · 2023-10-11T09:03:42Z

@pitrou @jorisvandenbossche any other concern here? I would like to merge it.

pitrou · 2023-10-11T09:05:13Z

@github-actions crossbow submit -g wheel -g python

pitrou · 2023-10-11T09:05:28Z

Let's make sure the 3.10 version change didn't break anything else?

github-actions · 2023-10-11T09:08:45Z

Revision: 27a979c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1c7e96dd5f

Task	Status
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest
test-conda-python-3.10-pandas-nightly
test-conda-python-3.10-spark-v3.5.0
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0
test-conda-python-3.8-spark-v3.5.0
test-conda-python-3.9
test-conda-python-3.9-pandas-latest
test-cuda-python
test-debian-11-python-3
test-fedora-35-python-3
test-ubuntu-20.04-python-3
test-ubuntu-22.04-python-3
wheel-clean
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-mojave-cp310-amd64
wheel-macos-mojave-cp311-amd64
wheel-macos-mojave-cp312-amd64
wheel-macos-mojave-cp38-amd64
wheel-macos-mojave-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2023-10-11T09:37:43Z

There are a lot of fsspec/s3fs errors that are unrelated to this PR.

test-conda-python-3.12 also fails installing dependencies from conda-forge, because gdb seems incompatible with Python 3.12. This can probably be fixed later.

pitrou · 2023-10-11T11:06:07Z

I think the CI results are ok otherwise, so we can merge.

jorisvandenbossche · 2023-10-11T11:19:37Z

A merge with main might be sufficient to get rid of the failures? We saw the same yesterday in the encryption PR, and after a merge they went away.
But indeed, if the triggered wheels builds are passing, that's probably also good enough to merge this directly.

jorisvandenbossche · 2023-10-11T11:25:54Z

Restarted one of the failing wheel builds, as that seemed a temporary download failure.

raulcd

👍 I am merging this! Thanks

### Rationale for this change Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release. ### What changes are included in this PR? This PR adds jobs to build pyarrow wheels for Python 3.12. ### Are these changes tested? They will be tested via archery tasks ### Are there any user-facing changes? No but users will be able to use pyarrow with Python 3.12 * Closes: #37880 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

conbench-apache-arrow · 2023-10-14T16:04:13Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 7695935.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

…pache#37901) ### Rationale for this change Python 3.12 will be released in the next couple of weeks. We should add the wheels for pyarrow on our 14.0.0 release. ### What changes are included in this PR? This PR adds jobs to build pyarrow wheels for Python 3.12. ### Are these changes tested? They will be tested via archery tasks ### Are there any user-facing changes? No but users will be able to use pyarrow with Python 3.12 * Closes: apache#37880 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

### Rationale for this change Like #43519 mentionies, now that the first `rc` is out, it's probably time to add CI coverage for Python 3.13 (and also start building wheels). ### What changes are included in this PR? I'm fairly new to the build/CI processes of the project, but I tried to follow the same template as #37901. I'll follow up afterwards with adding CI coverage for the free-threaded build as well. * GitHub Issue: #43519 Lead-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

github-actionsBot added Component: Python awaiting committer review Awaiting committer review labels Sep 27, 2023

This comment was marked as outdated.
Sign in to view

terencehonles mentioned this pull request Oct 2, 2023
GH-37880: [CI][Python][Packaging] Add support for Python 3.12 #37977
Closed

raulcd force-pushed the GH-37880 branch from e92c96b to b1d98eaCompare October 3, 2023 07:37

Use Python 3.10.11 version that contains the Macos .pkg
27a979c

github-actionsBot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 11, 2023

raulcd commented Oct 11, 2023
View reviewed changes

raulcd merged commit 7695935 into apache:mainOct 11, 2023

raulcd removed the awaiting changes Awaiting changes label Oct 11, 2023

github-actionsBot added the awaiting changes Awaiting changes label Oct 11, 2023

thehamish555 mentioned this pull request Oct 12, 2023
Can't install streamlit for Python 3.12 streamlit/streamlit#7506
Closed
4 tasks

jorisvandenbossche mentioned this pull request Aug 1, 2024
[Python][Packaging] Support Python 3.13 and upload wheels #43519
Closed
5 tasks

lysnikolaou mentioned this pull request Aug 2, 2024
GH-43519: [Python] Set up wheel building for Python 3.13 #43539
Merged

raulcd mentioned this pull request Aug 27, 2025
[Python][Packaging] Support Python 3.14 and upload wheels #47438
Closed
2 tasks

raulcd mentioned this pull request Jun 4, 2026
[Packaging][Python] Add support for Python 3.15 #50091
Open

Uh oh!

Conversation

raulcd commented Sep 27, 2023• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 27, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulcd commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

raulcd commented Sep 28, 2023

Uh oh!

github-actionsBot commented Sep 28, 2023

Uh oh!

raulcd commented Oct 3, 2023

Uh oh!

github-actionsBot commented Oct 3, 2023

Uh oh!

raulcd commented Oct 3, 2023

Uh oh!

raulcd commented Oct 10, 2023

Uh oh!

github-actionsBot commented Oct 10, 2023

Uh oh!

raulcd commented Oct 11, 2023

Uh oh!

pitrou commented Oct 11, 2023

Uh oh!

pitrou commented Oct 11, 2023

Uh oh!

github-actionsBot commented Oct 11, 2023

Uh oh!

pitrou commented Oct 11, 2023

Uh oh!

pitrou commented Oct 11, 2023

Uh oh!

jorisvandenbossche commented Oct 11, 2023

Uh oh!

jorisvandenbossche commented Oct 11, 2023

Uh oh!

raulcd commented Sep 27, 2023•
edited by github-actions Bot
Loading

raulcd commented Sep 27, 2023•
edited
Loading