GH-44071: [C++] Leak S3 structures if finalization happens too late

[pitrou]

Rationale for this change

Leaking S3 structures at shutdown can be better than inducing a segfault because those structures' destructors run too late at process exit.

This seems to avoid the crash when run under uwsgi in #44071

Are these changes tested?

Yes.

Are there any user-facing changes?

Hopefully not.

• GitHub Issue: [Python][S3] Segfault when using S3FileSystem in uwsgi #44071

[pitrou]

@github-actions crossbow submit -g cpp

[pitrou]

@github-actions crossbow submit wheelcp312*

[github-actions]

Revision: e7d9ea53df7292ec7a0016778d9679537f7ffd7e

Submitted crossbow builds: ursacomputing/crossbow @ actions-de41ec5070

Task	Status
r-binary-packages
test-r-arrow-backwards-compatibility
test-r-clang-sanitizer
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-extra-packages
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-install-local-minsizerel
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-macos-as-cran
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-release-latest
test-r-rocker-r-ver-latest
test-r-rstudio-r-base-4.1-opensuse155
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-r-sanitizer

[pitrou]

Need #44093 before we can properly test S3 on Windows wheel builds

Edit: it's now merged

[pitrou]

@github-actions crossbow submit wheelcp312*

[pitrou]

I notice that S3 is similarly untested on macOS Python builds: #44111

Edit: fixed now

[pitrou]

@github-actions crossbow submit wheel-macos*

[pitrou]

@github-actions crossbow submit -g cpp -g python -g wheel

[pitrou]

This is very hackish but solves the issue of using PyArrow under uwsgi. @felipecrv What is your opinion?

Edit: it did but it doesn't. I don't even understand what has changed since a couple hours ago... Now fixed hopefully.

[pitrou]

@github-actions crossbow submit -g cpp -g python -g wheel

[pitrou]

@github-actions crossbow submit wheelmacos wheellinux

[pitrou]

@github-actions crossbow submit -g python

[pitrou]

@github-actions crossbow submit wheelmacoscp312* wheellinuxcp312* wheelmacoscp313* wheellinuxcp313*

[pitrou]

@github-actions crossbow submit -g wheel

[pitrou]

@github-actions crossbow submit -g wheel -g python

[pitrou]

The C/GLib failure looks unrelated:
https://github.com/apache/arrow/actions/runs/10899317850/job/30247191830?pr=44090#step:9:270

../../c_glib/arrow-flight-glib/client.cpp:278: Warning: ArrowFlight: invalid "closure" annotation: only valid on callback parameters
../../c_glib/arrow-flight-glib/server.cpp:465: Warning: ArrowFlight: invalid "closure" annotation: only valid on callback parameters
<unknown>:: Fatal: ArrowFlight: warnings configured as fatal
<unknown>:: Fatal: ArrowFlight: warnings configured as fatal

What do you think @kou ?

[github-actions]

⚠️ GitHub issue #44071has been automatically assigned in GitHub to PR creator.

[pitrou]

@github-actions crossbow submit -g cpp

[pitrou]

@jorisvandenbossche Can you take a look at the Python test?

[github-actions]

Revision: 8946c1a9be668a1c53df9e61c75119553c009be7

Submitted crossbow builds: ursacomputing/crossbow @ actions-60301353eb

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14

[kou]

The C/GLib failure looks unrelated: https://github.com/apache/arrow/actions/runs/10899317850/job/30247191830?pr=44090#step:9:270

../../c_glib/arrow-flight-glib/client.cpp:278: Warning: ArrowFlight: invalid "closure" annotation: only valid on callback parameters
../../c_glib/arrow-flight-glib/server.cpp:465: Warning: ArrowFlight: invalid "closure" annotation: only valid on callback parameters
<unknown>:: Fatal: ArrowFlight: warnings configured as fatal
<unknown>:: Fatal: ArrowFlight: warnings configured as fatal

What do you think @kou ?

It's unrelated. I'll fix it by #44153.

[kou]

Fixed.

[pitrou]

@github-actions crossbow submit wheelmacos wheellinux

[pitrou]

@github-actions crossbow submit -g python

[github-actions]

Revision: 07cc766

Submitted crossbow builds: ursacomputing/crossbow @ actions-4f61f433c5

Task	Status
wheel-macos-monterey-cp310-cp310-amd64
wheel-macos-monterey-cp310-cp310-arm64
wheel-macos-monterey-cp311-cp311-amd64
wheel-macos-monterey-cp311-cp311-arm64
wheel-macos-monterey-cp312-cp312-amd64
wheel-macos-monterey-cp312-cp312-arm64
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-macos-monterey-cp39-cp39-amd64
wheel-macos-monterey-cp39-cp39-arm64
wheel-manylinux-2-28-cp310-cp310-amd64
wheel-manylinux-2-28-cp310-cp310-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-manylinux-2-28-cp312-cp312-amd64
wheel-manylinux-2-28-cp312-cp312-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-manylinux-2-28-cp39-cp39-amd64
wheel-manylinux-2-28-cp39-cp39-arm64
wheel-manylinux-2014-cp310-cp310-amd64
wheel-manylinux-2014-cp310-cp310-arm64
wheel-manylinux-2014-cp311-cp311-amd64
wheel-manylinux-2014-cp311-cp311-arm64
wheel-manylinux-2014-cp312-cp312-amd64
wheel-manylinux-2014-cp312-cp312-arm64
wheel-manylinux-2014-cp313-cp313-amd64
wheel-manylinux-2014-cp313-cp313-arm64
wheel-manylinux-2014-cp313-cp313t-amd64
wheel-manylinux-2014-cp313-cp313t-arm64
wheel-manylinux-2014-cp39-cp39-amd64
wheel-manylinux-2014-cp39-cp39-arm64

[github-actions]

Revision: 07cc766

Submitted crossbow builds: ursacomputing/crossbow @ actions-7df153a302

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

[pitrou]

CI is green. @jorisvandenbossche Are you available to take a quick look at the Python test?

[jorisvandenbossche]

Are you available to take a quick look at the Python test?

Looks good!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 0f7b5e5.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 9 possible false positives for unstable benchmarks that are known to sometimes produce them.