[Python] a segfault on `import pyarrow` on MacOS 11.6 after pip update to pyarrow version 16.1.0

[ianna]

Describe the bug, including details regarding any error messages, version, and platform.

Hi,

I running into an issue with a newly released 16.1.0 version. The 16.0.0 works without issues, but updating to 16.1.0 causes a segfault. Any help is greatly appreciated. Thanks!

• Python 3.11.5
• Clang 14.0.6
• MacOS 11.6 (x86_64)

 % conda list
# packages in environment at /Users/yana/anaconda3/envs/pyarrow-test:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl  
bottleneck                1.3.7           py311hb3a5e46_0  
bzip2                     1.0.8                h6c40b1e_6  
ca-certificates           2024.3.11            hecd8cb5_0  
intel-openmp              2023.1.0         ha357a0b_43548  
libcxx                    14.0.6               h9765a3e_0  
libffi                    3.4.4                hecd8cb5_1  
mkl                       2023.1.0         h8e150cf_43560  
mkl-service               2.4.0           py311h6c40b1e_1  
mkl_fft                   1.3.8           py311h6c40b1e_0  
mkl_random                1.2.4           py311ha357a0b_0  
ncurses                   6.4                  hcec6c5f_0  
numexpr                   2.8.7           py311h728a8a3_0  
numpy                     1.26.4          py311h728a8a3_0  
numpy-base                1.26.4          py311h53bf9ac_0  
openssl                   3.0.13               hca72f7f_1  
pandas                    2.2.1           py311hdb55bb0_0  
pip                       24.0            py311hecd8cb5_0  
pyarrow                   16.1.0                   pypi_0    pypi
python                    3.11.5               hf27a42d_0  
python-dateutil           2.9.0post0      py311hecd8cb5_0  
python-tzdata             2023.3             pyhd3eb1b0_0  
pytz                      2024.1          py311hecd8cb5_0  
readline                  8.2                  hca72f7f_0  
setuptools                69.5.1          py311hecd8cb5_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.45.3               h6c40b1e_0  
tbb                       2021.8.0             ha357a0b_0  
tk                        8.6.14               h4d00af3_0  
tzdata                    2024a                h04d1e81_0  
wheel                     0.43.0          py311hecd8cb5_0  
xz                        5.4.6                h6c40b1e_1  
zlib                      1.2.13               h4b97444_1  

% python
Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
zsh: segmentation fault  python

The test_pyarrow.py has only one line: import pyarrow:

% lldb python test_pyarrow.py         
(lldb) target create "python"
Current executable set to '/Users/yana/anaconda3/bin/python' (x86_64).
(lldb) settings set -- target.run-args  "test_pyarrow.py"
(lldb) run
Process 35199 launched: '/Users/yana/anaconda3/bin/python' (x86_64)
Process 35199 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x0000000129c8685e libarrow.1601.dylib`malloc_conf_init_helper + 302
libarrow.1601.dylib`malloc_conf_init_helper:
->  0x129c8685e <+302>: movzbl (%rbx), %eax
    0x129c86861 <+305>: testb  %al, %al
    0x129c86863 <+307>: je     0x129c8936e               ; <+11326>
    0x129c86869 <+313>: movq   %rbx, %rcx
(lldb) exit

Component(s)

Python, Release

[jorisvandenbossche]

@ianna thanks for the report!

The crash seems to be related to jemalloc. Not a solution, but just curious: could you set ARROW_DEFAULT_MEMORY_POOL=system env variable before importing pyarrow?

[ianna]

@ianna thanks for the report!

The crash seems to be related to jemalloc. Not a solution, but just curious: could you set ARROW_DEFAULT_MEMORY_POOL=system env variable before importing pyarrow?

@jorisvandenbossche - thanks for prompt reply! Setting the env variable does not help :-(

% python
Python 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.environ["ARROW_DEFAULT_MEMORY_POOL"])
system
>>> import pyarrow
zsh: segmentation fault  python

[paleolimbot]

I can reproduce on MacOS x86_64 with pip install pyarrow and import pyarrow. The lldb backtrace I get is:

(lldb) bt all
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x00000001077a685e libarrow.1601.dylib`malloc_conf_init_helper + 302
    frame #1: 0x00000001077a6207 libarrow.1601.dylib`malloc_init_hard_a0_locked + 135
    frame #2: 0x00000001077a9c1f libarrow.1601.dylib`malloc_init_hard + 159
    frame #3: 0x00000001006bbb47 dyld`ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 535
    frame #4: 0x00000001006bbf52 dyld`ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
    frame #5: 0x00000001006b6ae6 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 492
    frame #6: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #7: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #8: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #9: 0x00000001006b6a51 dyld`ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 343
    frame #10: 0x00000001006b489f dyld`ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 191
    frame #11: 0x00000001006b4940 dyld`ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
    frame #12: 0x00000001006a4a12 dyld`dyld::runInitializers(ImageLoader*) + 82
    frame #13: 0x00000001006b011a dyld`dlopen_internal + 616
    frame #14: 0x00007fff20406c94 libdyld.dylib`dlopen_internal(char const*, int, void*) + 185
    frame #15: 0x00007fff203f507e libdyld.dylib`dlopen + 28
    frame #16: 0x0000000100244ff5 python`_PyImport_LoadDynamicModuleWithSpec + 693
    frame #17: 0x0000000100244337 python`_imp_create_dynamic + 167
    frame #18: 0x00000001000e84fa python`cfunction_vectorcall_FASTCALL + 106
    frame #19: 0x00000001001f6175 python`_PyEval_EvalFrameDefault + 284165
    frame #20: 0x000000010007bec9 python`_PyObject_VectorcallTstate.779 + 73
    frame #21: 0x0000000100080c1e python`object_vacall + 414
    frame #22: 0x0000000100080a2a python`PyObject_CallMethodObjArgs + 234
    frame #23: 0x0000000100242262 python`PyImport_ImportModuleLevelObject + 3490
    frame #24: 0x00000001001e3b4a python`_PyEval_EvalFrameDefault + 208858
    frame #25: 0x00000001001ae2bd python`PyEval_EvalCode + 253
    frame #26: 0x00000001001a9a55 python`builtin_exec + 437
    frame #27: 0x00000001000e8462 python`cfunction_vectorcall_FASTCALL_KEYWORDS + 98
    frame #28: 0x00000001001f6175 python`_PyEval_EvalFrameDefault + 284165
    frame #29: 0x000000010007bec9 python`_PyObject_VectorcallTstate.779 + 73
    frame #30: 0x0000000100080c1e python`object_vacall + 414
    frame #31: 0x0000000100080a2a python`PyObject_CallMethodObjArgs + 234
    frame #32: 0x0000000100242262 python`PyImport_ImportModuleLevelObject + 3490
    frame #33: 0x00000001001e3b4a python`_PyEval_EvalFrameDefault + 208858
    frame #34: 0x00000001001ae2bd python`PyEval_EvalCode + 253
    frame #35: 0x0000000100277893 python`run_mod + 275
    frame #36: 0x0000000100277c80 python`PyRun_InteractiveOneObjectEx + 592
    frame #37: 0x0000000100276e1d python`_PyRun_InteractiveLoopObject + 141
    frame #38: 0x00000001002769bf python`_PyRun_AnyFileObject + 63
    frame #39: 0x0000000100279cca python`PyRun_AnyFileExFlags + 58
    frame #40: 0x00000001002a333f python`pymain_run_stdin + 175
    frame #41: 0x00000001002a294d python`Py_RunMain + 637
    frame #42: 0x0000000100001a68 python`main + 56
    frame #43: 0x00007fff20404f3d libdyld.dylib`start + 1
  thread #2
    frame #0: 0x00007fff203b593e libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff203e64c1 libsystem_pthread.dylib`_pthread_wqthread + 414
    frame #2: 0x00007fff203e542f libsystem_pthread.dylib`start_wqthread + 15
  thread #3
    frame #0: 0x00007fff203b593e libsystem_kernel.dylib`__workq_kernreturn + 10
    frame #1: 0x00007fff203e64c1 libsystem_pthread.dylib`_pthread_wqthread + 414
    frame #2: 0x00007fff203e542f libsystem_pthread.dylib`start_wqthread + 15
(lldb) 

[gidler]

I have the same EXC_BAD_ACCESS issue and lldb backtrace on MacOS 11.7.4 (x86_64) with python 3.11.0.

pip install pyarrow==16.0.0 does not have the issue.

[paleolimbot]

I did a quick google of the stack trace, and there's a note in jemalloc specifically about the function that crashes (may or may not be related):

https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L229-L245

Reproduced:

	 * If another constructor in the same binary is using mallctl to e.g.
	 * set up extent hooks, it may end up running before this one, and
	 * malloc_init_hard will crash trying to lock the uninitialized lock. So
	 * we force an initialization of the lock in malloc_init_hard as well.

Some links to possible places referred to by the stack trace:

• malloc_conf_init_helper(): https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L1084
• Call to malloc_conf_init_helper() in malloc_conf_ini(): https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L1795
• Call to malloc_conf_init() in malloc_init_hard_a0_locked(): https://github.com/jemalloc/jemalloc/blob/5afff2e44e8d31ef1e9eb01d6b1327fe111835ed/src/jemalloc.c#L1863

[jorisvandenbossche]

The diff between 16.0.0 and 16.1.0 is here: apache-arrow-16.0.0...apache-arrow-16.1.0.

The only change that seems somewhat related is #41567 (unless the fixes for scalar scratch space fix for some reason caused this), although this is also strange because in the past this macos-latest should also have defaulted to macos-13 at some point, I assume?

cc @raulcd