GH-38183: [CI][Python] Use pipx to install GCS testbench by pitrou · Pull Request #43852 · apache/arrow

pitrou · 2024-08-27T15:11:10Z

Rationale for this change

Installing the GCS testbench using the same Python that's being used to test PyArrow is fragile: some testbench versions may not be compatible, or there could be conflicts among the dependencies of the respective libraries.

What changes are included in this PR?

Use pipx to install the GCS testbench in a separate, controlled environment, using an appropriate Python version.

Are these changes tested?

Yes, by CI.

Are there any user-facing changes?

No.

GitHub Issue: [CI][Python] Use pipx to install GCS testbench in order to decouple version of Python under test #38183

github-actions · 2024-08-27T15:11:40Z

⚠️ GitHub issue #38183has been automatically assigned in GitHub to PR creator.

pitrou · 2024-08-27T15:51:47Z

@github-actions crossbow submit -g cpp

pitrou · 2024-08-27T15:52:00Z

@github-actions crossbow submit -g python -g wheel

pitrou · 2024-08-27T16:55:33Z

@github-actions crossbow submit wheel-windows-cp310-amd64

pitrou · 2024-08-27T18:31:36Z

@github-actions crossbow submit wheel-windows-cp310-amd64 wheel-windows-cp313-amd64

pitrou · 2024-08-27T18:31:48Z

@github-actions crossbow submit -g python -g cpp

pitrou · 2024-08-28T08:46:49Z

@github-actions crossbow submit wheel-windows*

github-actions · 2024-08-28T17:10:04Z

Revision: 1da4ed8

Submitted crossbow builds: ursacomputing/crossbow @ actions-7287ca77dd

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp313-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp313-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp313-amd64
wheel-manylinux-2-28-cp313-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp313-amd64
wheel-manylinux-2014-cp313-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp313-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2024-08-28T17:21:02Z

@github-actions crossbow submit wheel-windows*

pitrou · 2024-08-28T20:07:28Z

@github-actions crossbow submit wheel-windows*

github-actions · 2024-08-28T20:10:13Z

Revision: 55ece7b

Submitted crossbow builds: ursacomputing/crossbow @ actions-4c656a1090

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp313-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2024-08-28T21:11:41Z

@github-actions crossbow submit -g python -g cpp

pitrou · 2024-08-28T21:11:52Z

@kou Could you take a look at this PR?

github-actions · 2024-08-28T21:14:35Z

Revision: 55ece7b

Submitted crossbow builds: ursacomputing/crossbow @ actions-2df46cee67

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-1.26
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-pandas-nightly-numpy-nightly
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.8
test-conda-python-3.8-pandas-1.0-numpy-1.19
test-conda-python-3.9
test-conda-python-3.9-pandas-latest-numpy-latest
test-conda-python-emscripten
test-cuda-cpp
test-cuda-python
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-cpp
test-fedora-39-python-3
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-20.04-python-3
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-22.04-python-3
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14

kou

+1

test-conda-python-emscripten failure is unrelated: #43868

jorisvandenbossche · 2024-08-29T07:41:21Z

+
 # This script is run with PYTHON undefined in some places,
 # but those only use older pythons.
 if [[ -z "${PYTHON_VERSION}" ]] || [[ "${PYTHON_VERSION}" != "3.13" ]]; then


This if check could be removed now? (AFAIK we only added it because we couldn't install grpcio on python 3.13, but now this is using pipx (with fixed python version) it shouldn't matter there are no python 3.13 wheels yet?

Yes, I should open a followup issue.

Opened #43883

conbench-apache-arrow · 2024-08-29T08:35:19Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 58415d1.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 135 possible false positives for unstable benchmarks that are known to sometimes produce them.

…e#43852) ### Rationale for this change Installing the GCS testbench using the same Python that's being used to test PyArrow is fragile: some testbench versions may not be compatible, or there could be conflicts among the dependencies of the respective libraries. ### What changes are included in this PR? Use `pipx` to install the GCS testbench in a separate, controlled environment, using an appropriate Python version. ### Are these changes tested? Yes, by CI. ### Are there any user-facing changes? No. * GitHub Issue: apache#38183 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actionsBot added Component: R Component: C++ Component: Python awaiting review Awaiting review labels Aug 27, 2024

pitrou force-pushed the gh38183-pipx branch from 1d9c90f to 2148b25Compare August 27, 2024 15:51

This comment was marked as outdated.
Sign in to view

pitrou force-pushed the gh38183-pipx branch 4 times, most recently from 4d3e1cb to 119363cCompare August 27, 2024 17:57

This comment was marked as outdated.
Sign in to view

pitrou added 8 commits August 28, 2024 10:26

apacheGH-38183: [CI][Python] Use pipx to install GCS testbench
f49f84c

Try to fix AppVeyor and MinGW builds
8b7bf3c

Further try to fix CI issues
2c1c742

Try to debug
4852e0d

Another attempt
de650c5

Make sure the installed Python is used
fbb0c77

Try to fix Windows wheel test
2663881

Try to fix remaining CI failures
ecc8be2

pitrou force-pushed the gh38183-pipx branch from 2e472bf to ecc8be2Compare August 28, 2024 08:46

pitrou requested review from kou, raulcd and thisisnic as code owners August 28, 2024 17:07

Install pip and setuptools during run, not build
d194432

This comment was marked as outdated.
Sign in to view

Use the py launcher instead of trying to mess with PATH
55ece7b

kou approved these changes Aug 29, 2024
View reviewed changes

kou merged commit 58415d1 into apache:mainAug 29, 2024

kou removed the awaiting review Awaiting review label Aug 29, 2024

kou mentioned this pull request Aug 29, 2024
[CI][Python] Use pipx to install GCS testbench in order to decouple version of Python under test #38183
Closed

github-actionsBot added the awaiting merge Awaiting merge label Aug 29, 2024

pitrou deleted the gh38183-pipx branch August 29, 2024 06:35

This was referenced Aug 29, 2024
[C++][CI] Python grpcio built from source on ARM64 Ubuntu 20.04 build #43847
Closed
GH-43847: [CI][C++][Python] Try to trigger binary installation of grpcio #43848
Closed

jorisvandenbossche reviewed Aug 29, 2024
View reviewed changes

github-actionsBot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Aug 29, 2024

pitrou mentioned this pull request Aug 29, 2024
[CI] Remove Python version guard when installing GCS testbench #43883
Closed

pitrou mentioned this pull request Sep 16, 2024
[CI][Python][C++] Avoid compiling grpcio from source #33630
Closed

jorisvandenbossche mentioned this pull request Sep 23, 2024
[Python][Packaging] Support Python 3.13 and upload wheels #43519
Closed
5 tasks

Uh oh!

Conversation

pitrou commented Aug 27, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Aug 27, 2024

Uh oh!

pitrou commented Aug 27, 2024

Uh oh!

pitrou commented Aug 27, 2024

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Aug 27, 2024

Uh oh!

This comment was marked as outdated.

pitrou commented Aug 27, 2024

Uh oh!

pitrou commented Aug 27, 2024

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

pitrou commented Aug 28, 2024

Uh oh!

github-actionsBot commented Aug 28, 2024

Uh oh!

pitrou commented Aug 28, 2024

Uh oh!

This comment was marked as outdated.

pitrou commented Aug 28, 2024

Uh oh!

github-actionsBot commented Aug 28, 2024

Uh oh!

pitrou commented Aug 28, 2024

Uh oh!

pitrou commented Aug 28, 2024

Uh oh!

github-actionsBot commented Aug 28, 2024

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

jorisvandenbosscheAug 29, 2024

Choose a reason for hiding this comment

Uh oh!

pitrouAug 29, 2024

Choose a reason for hiding this comment

Uh oh!

pitrouAug 29, 2024

Choose a reason for hiding this comment

Uh oh!

conbench-apache-arrowBot commented Aug 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pitrou commented Aug 27, 2024•
edited
Loading