[Python] UWSGI test is crashed with AWS SDK C++ for C++ 1.11.587

[kou]

Describe the bug, including details regarding any error messages, version, and platform.

#45306 upgraded bundled AWS SDK C++ for C++ to 1.11.587 from 1.10.55.

We have a workaround for UWSGI + S3 filesystem use: #44090

It worked with 1.10.55 but it doesn't work with 1.11.587.

#45306 disabled the UWSGI integration test. We should re-enable it.

Component(s)

Continuous Integration, Python

[pitrou]

Can you mention on which CI build(s) the test crashed?

[kou]

We don't have any crashed CI build for now because #45306 added a change that skips the test.

We need to re-enable the test to show the crash:

diff --git a/python/pyarrow/tests/test_fs.py b/python/pyarrow/tests/test_fs.py
index e90613a662..2838372bd1 100644
--- a/python/pyarrow/tests/test_fs.py
+++ b/python/pyarrow/tests/test_fs.py
@@ -2043,8 +2043,6 @@ def test_concurrent_s3fs_init():
 
 
 @pytest.mark.s3
-@pytest.mark.skip(reason="atexit(ensure_s3_finalized) will be called too late "
- "with bundled aws-sdk-cpp 1.11.587")
 @pytest.mark.skipif(running_on_musllinux(), reason="Leaking S3ClientFinalizer causes "
 "segfault on musl based systems")
 def test_uwsgi_integration():

[pitrou]

I know, but we should have a CI build that has failed at some point. Can you link to the failing build?

[kou]

https://github.com/ursacomputing/crossbow/actions/runs/15573108643/job/43852922506#step:6:12518

____________________________ test_uwsgi_integration ____________________________

 @pytest.mark.s3
 @pytest.mark.skipif(running_on_musllinux(), reason="Leaking S3ClientFinalizer causes "
 "segfault on musl based systems")
 def test_uwsgi_integration():
 # GH-44071: using S3FileSystem under uwsgi shouldn't lead to a crash at shutdown
 try:
 subprocess.check_call(["uwsgi", "--version"])
 except FileNotFoundError:
 pytest.skip("uwsgi not installed on this Python")
 
 port = find_free_port()
 args = ["uwsgi", "-i", "--http", f"127.0.0.1:{port}",
 "--wsgi-file", os.path.join(here, "wsgi_examples.py")]
 proc = subprocess.Popen(args, stdin=subprocess.DEVNULL)
 # Try to fetch URL, it should return 200 Ok...
 try:
 url = f"http://127.0.0.1:{port}/s3/"
 start_time = time.time()
 error = None
 while time.time() < start_time + 5:
 try:
 with urlopen(url) as resp:
 assert resp.status == 200
 break
 except OSError as e:
 error = e
 time.sleep(0.1)
 else:
 pytest.fail(f"Could not fetch {url!r}: {error}")
 finally:
 proc.terminate()
 # ... and uwsgi should gracefully shutdown after it's been asked above
> assert proc.wait() == 30 # UWSGI_END_CODE = 30
 ^^^^^^^^^^^^^^^^^^^^^^^^
E AssertionError: assert -6 == 30
E + where -6 = wait()
E + where wait = <Popen: returncode: -6 args: ['uwsgi', '-i', '--http', '127.0.0.1:35863', '-...>.wait

arrow-dev/lib/python3.11/site-packages/pyarrow/tests/test_fs.py:2077: AssertionError
----------------------------- Captured stdout call -----------------------------
2.0.29
----------------------------- Captured stderr call -----------------------------
*** Starting uWSGI 2.0.29 (64bit) on [Wed Jun 11 00:55:57 2025] ***
compiled with version: 10.2.1 20210130 (Red Hat 10.2.1-11) on 26 April 2025 16:59:37
os: Linux-6.11.0-1015-azure #15~24.04.1-Ubuntu SMP Thu May 1 02:52:08 UTC 2025
nodename: f2471d7e28bf
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /
detected binary path: /usr/bin/python3.11
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 127.0.0.1:35863 fd 4
spawned uWSGI http 1 (pid: 23144)
uwsgi socket 0 bound to TCP address 127.0.0.1:37629 (port auto-assigned) fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
Python version: 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0]
--- Python VM already initialized ---
Python main interpreter initialized at 0xa840f8
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72904 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0xa840f8 pid: 23143 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
spawned uWSGI worker 1 (and the only) (pid: 23143, cores: 1)
[pid: 23143|app: 0|req: 1/1] 127.0.0.1 () {30 vars in 347 bytes} [Wed Jun 11 00:55:58 2025] GET /s3/ => generated 12 bytes in 31 msecs (HTTP/1.1 200) 1 headers in 44 bytes (1 switches on core 0)
/arrow/cpp/src/arrow/filesystem/s3fs.cc:3492: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit
Exception ignored in atexit callback: <cyfunction ensure_s3_finalized at 0x7f7d0475cad0>
Traceback (most recent call last):
 File "pyarrow/_s3fs.pyx", line 75, in pyarrow._s3fs.ensure_s3_finalized
 File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: FinalizeS3 called too late
pure virtual method called
terminate called without an active exception