Uh oh!
There was an error while loading. Please reload this page.
GH-15054: [Python] Finalize S3 on pytest exit to prevent shutdown crashes - #33793
GH-15054: [Python] Finalize S3 on pytest exit to prevent shutdown crashes#33793westonpace wants to merge 2 commits into
Conversation
lidavidm
commented
Jan 19, 2023
In a wheel though, we statically link everything, and then destruction order is undefined as usual? |
| doctest_namespace["local_path"] = str(tmp_path) | ||
| doctest_namespace["path"] = str(path) | ||
| yield | ||
There was a problem hiding this comment.
I think the linter error is only because it expects two empty lines here.
westonpace
commented
Jan 23, 2023
Yes, I think that could explain it. Still, this fix would require pyarrow users to know about and remember calling the finalize S3 method. I'm going to pause this idea and make an attempt at an arrow::Shutdown. |
westonpace
commented
Jan 25, 2023
Closing in favor of #33858 |
For some reason destroying an S3Client at shutdown can cause a crash in our tests. It appears to be because S3's logging is already shut down. I wouldn't think this would happen because I would think the linker would tear down S3 shared libraries AFTER it tears down Arrow shared libraries.
However, we can also finalize S3 in conftest.py to prevent this crash.
It would still be good to understand the root cause. Otherwise we need to advise our users to call finalize_s3.