[Python] a segfault on `import pyarrow` on MacOS 11.6 after pip update to pyarrow version 16.1.0

[ianna]

Describe the bug, including details regarding any error messages, version, and platform.

Hi,

I running into an issue with a newly released 16.1.0 version. The 16.0.0 works without issues, but updating to 16.1.0 causes a segfault. Any help is greatly appreciated. Thanks!

• Python 3.11.5
• Clang 14.0.6
• MacOS 11.6 (x86_64)

 % conda list
# packages in environment at /Users/yana/anaconda3/envs/pyarrow-test:
#
# Name Version Build Channel
blas 1.0 mkl 
bottleneck 1.3.7 py311hb3a5e46_0 
bzip2 1.0.8 h6c40b1e_6 
ca-certificates 2024.3.11 hecd8cb5_0 
intel-openmp 2023.1.0 ha357a0b_43548 
libcxx 14.0.6 h9765a3e_0 
libffi 3.4.4 hecd8cb5_1 
mkl 2023.1.0 h8e150cf_43560 
mkl-service 2.4.0 py311h6c40b1e_1 
mkl_fft 1.3.8 py311h6c40b1e_0 
mkl_random 1.2.4 py311ha357a0b_0 
ncurses 6.4 hcec6c5f_0 
numexpr 2.8.7 py311h728a8a3_0 
numpy 1.26.4 py311h728a8a3_0 
numpy-base 1.26.4 py311h53bf9ac_0 
openssl 3.0.13 hca72f7f_1 
pandas 2.2.1 py311hdb55bb0_0 
pip 24.0 py311hecd8cb5_0 
pyarrow 16.1.0 pypi_0 pypi
python 3.11.5 hf27a42d_0 
python-dateutil 2.9.0post0 py311hecd8cb5_0 
python-tzdata 2023.3 pyhd3eb1b0_0 
pytz 2024.1 py311hecd8cb5_0 
readline 8.2 hca72f7f_0 
setuptools 69.5.1 py311hecd8cb5_0 
six 1.16.0 pyhd3eb1b0_1 
sqlite 3.45.3 h6c40b1e_0 
tbb 2021.8.0 ha357a0b_0 
tk 8.6.14 h4d00af3_0 
tzdata 2024a h04d1e81_0 
wheel 0.43.0 py311hecd8cb5_0 
xz 5.4.6 h6c40b1e_1 
zlib 1.2.13 h4b97444_1 

% python
Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
zsh: segmentation fault python

The test_pyarrow.py has only one line: import pyarrow:

% lldb python test_pyarrow.py 
(lldb) target create "python"
Current executable set to '/Users/yana/anaconda3/bin/python' (x86_64).
(lldb) settings set -- target.run-args "test_pyarrow.py"
(lldb) run
Process 35199 launched: '/Users/yana/anaconda3/bin/python' (x86_64)
Process 35199 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
 frame #0: 0x0000000129c8685e libarrow.1601.dylib`malloc_conf_init_helper + 302
libarrow.1601.dylib`malloc_conf_init_helper:
-> 0x129c8685e <+302>: movzbl (%rbx), %eax
 0x129c86861 <+305>: testb %al, %al
 0x129c86863 <+307>: je 0x129c8936e ; <+11326>
 0x129c86869 <+313>: movq %rbx, %rcx
(lldb) exit

Component(s)

Python, Release

[jorisvandenbossche]

@ianna thanks for the report!

The crash seems to be related to jemalloc. Not a solution, but just curious: could you set ARROW_DEFAULT_MEMORY_POOL=system env variable before importing pyarrow?

[ianna]

@ianna thanks for the report!

The crash seems to be related to jemalloc. Not a solution, but just curious: could you set ARROW_DEFAULT_MEMORY_POOL=system env variable before importing pyarrow?

@jorisvandenbossche - thanks for prompt reply! Setting the env variable does not help :-(

% python
Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.environ["ARROW_DEFAULT_MEMORY_POOL"])
system
>>> import pyarrow
zsh: segmentation fault python

[paleolimbot]

I can reproduce on MacOS x86_64 with pip install pyarrow and import pyarrow. The lldb backtrace I get is:

(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
 * frame #0: 0x00000001077a685e libarrow.1601.dylib`malloc_conf_init_helper + 302
 frame #1: 0x00000001077a6207 libarrow.1601.dylib`malloc_init_hard_a0_locked + 135
 frame #2: 0x00000001077a9c1f libarrow.1601.dylib`malloc_init_hard + 159
 frame #3: 0x00000001006bbb47 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 535
 frame #4: 0x00000001006bbf52 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
 frame #5: 0x00000001006b6ae6 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 492
 frame #6: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
 frame #7: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
 frame #8: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
 frame #9: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
 frame #10: 0x00000001006b489f dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 191
 frame #11: 0x00000001006b4940 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
 frame #12: 0x00000001006a4a12 dyld`dyld::runInitializers(ImageLoader*) + 82
 frame #13: 0x00000001006b011a dyld`dlopen_internal + 616
 frame #14: 0x00007fff20406c94 libdyld.dylib`dlopen_internal(char const*, int, void*) + 185
 frame #15: 0x00007fff203f507e libdyld.dylib`dlopen + 28
 frame #16: 0x0000000100244ff5 python`_PyImport_LoadDynamicModuleWithSpec + 693
 frame #17: 0x0000000100244337 python`_imp_create_dynamic + 167
 frame #18: 0x00000001000e84fa python`cfunction_vectorcall_FASTCALL + 106
 frame #19: 0x00000001001f6175 python`_PyEval_EvalFrameDefault + 284165
 frame #20: 0x000000010007bec9 python`_PyObject_VectorcallTstate.779 + 73
 frame #21: 0x0000000100080c1e python`object_vacall + 414
 frame #22: 0x0000000100080a2a python`PyObject_CallMethodObjArgs + 234
 frame #23: 0x0000000100242262 python`PyImport_ImportModuleLevelObject + 3490
 frame #24: 0x00000001001e3b4a python`_PyEval_EvalFrameDefault + 208858
 frame #25: 0x00000001001ae2bd python`PyEval_EvalCode + 253
 frame #26: 0x00000001001a9a55 python`builtin_exec + 437
 frame #27: 0x00000001000e8462 python`cfunction_vectorcall_FASTCALL_KEYWORDS + 98
 frame #28: 0x00000001001f6175 python`_PyEval_EvalFrameDefault + 284165
 frame #29: 0x000000010007bec9 python`_PyObject_VectorcallTstate.779 + 73
 frame #30: 0x0000000100080c1e python`object_vacall + 414
 frame #31: 0x0000000100080a2a python`PyObject_CallMethodObjArgs + 234
 frame #32: 0x0000000100242262 python`PyImport_ImportModuleLevelObject + 3490
 frame #33: 0x00000001001e3b4a python`_PyEval_EvalFrameDefault + 208858
 frame #34: 0x00000001001ae2bd python`PyEval_EvalCode + 253
 frame #35: 0x0000000100277893 python`run_mod + 275
 frame #36: 0x0000000100277c80 python`PyRun_InteractiveOneObjectEx + 592
 frame #37: 0x0000000100276e1d python`_PyRun_InteractiveLoopObject + 141
 frame #38: 0x00000001002769bf python`_PyRun_AnyFileObject + 63
 frame #39: 0x0000000100279cca python`PyRun_AnyFileExFlags + 58
 frame #40: 0x00000001002a333f python`pymain_run_stdin + 175
 frame #41: 0x00000001002a294d python`Py_RunMain + 637
 frame #42: 0x0000000100001a68 python`main + 56
 frame #43: 0x00007fff20404f3d libdyld.dylib`start + 1
 thread #2
 frame #0: 0x00007fff203b593e libsystem_kernel.dylib`__workq_kernreturn + 10
 frame #1: 0x00007fff203e64c1 libsystem_pthread.dylib`_pthread_wqthread + 414
 frame #2: 0x00007fff203e542f libsystem_pthread.dylib`start_wqthread + 15
 thread #3
 frame #0: 0x00007fff203b593e libsystem_kernel.dylib`__workq_kernreturn + 10
 frame #1: 0x00007fff203e64c1 libsystem_pthread.dylib`_pthread_wqthread + 414
 frame #2: 0x00007fff203e542f libsystem_pthread.dylib`start_wqthread + 15
(lldb) 

[gidler]

I have the same EXC_BAD_ACCESS issue and lldb backtrace on MacOS 11.7.4 (x86_64) with python 3.11.0.

pip install pyarrow==16.0.0 does not have the issue.

[paleolimbot]

I did a quick google of the stack trace, and there's a note in jemalloc specifically about the function that crashes (may or may not be related):

https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L229-L245

Reproduced:

	 * If another constructor in the same binary is using mallctl to e.g.
	 * set up extent hooks, it may end up running before this one, and
	 * malloc_init_hard will crash trying to lock the uninitialized lock. So
	 * we force an initialization of the lock in malloc_init_hard as well.

Some links to possible places referred to by the stack trace:

• malloc_conf_init_helper(): https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L1084
• Call to malloc_conf_init_helper() in malloc_conf_ini(): https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L1795
• Call to malloc_conf_init() in malloc_init_hard_a0_locked(): https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L1863

[jorisvandenbossche]

The diff between 16.0.0 and 16.1.0 is here: apache-arrow-16.0.0...apache-arrow-16.1.0.

The only change that seems somewhat related is #41567 (unless the fixes for scalar scratch space fix for some reason caused this), although this is also strange because in the past this macos-latest should also have defaulted to macos-13 at some point, I assume?

cc @raulcd