importing pyarrow and Tensorflow crashes

[brills]

Hi,

I'm not sure if this is resolved by the recent change to the linker script [1] as I browsed several related issues / PRs [2][3][4]. But currently with arrow 0.13.0 and tensorflow 1.13 or nightly (the "tf-nightly" pip package), if pyarrow is imported before tensorflow is, python crashes:

To reproduce:

To repreduce:

$ pip install tf-nightly
$ pip install pyarrow==0.13.0

$ python

import pyarrow
import tensorflow

216726 segmentation fault python

[1] 0289af2#diff-3c274588934dfc0526783b59d1f4fff2
[2] #1953
[3] #2096
[4] #2218

Stack trace at crash:

#0  0x0000000000000000 in ?? ()                                                                                                    
#1  0x00007ff492802739 in __pthread_once_slow (                                                                                    
    once_control=0x7ff462f00e88 <tensorflow::port::(anonymous namespace)::cpuid_once_flag>,                                        
    init_routine=0x7ff46f62dd60 <__once_proxy>) at pthread_once.c:116
#2  0x00007ff46251eeaa in void std::call_once<void (&)()>(std::once_flag&, void (&)()) ()                                          
   from /usr/local/google/home/zhuo/venv/en/local/lib/python2.7/site-packages/tensorflow/python/../libtensorflow_framework.so.1
#3  0x00007ff46251eede in tensorflow::port::TestCPUFeature(tensorflow::port::CPUFeature) ()
   from /usr/local/google/home/zhuo/venv/en/local/lib/python2.7/site-packages/tensorflow/python/../libtensorflow_framework.so.1
#4  0x00007ff46251edf1 in tensorflow::port::(anonymous namespace)::CheckFeatureOrDie(tensorflow::port::CPUFeature, std::string const
&) () from /usr/local/google/home/zhuo/venv/en/local/lib/python2.7/site-packages/tensorflow/python/../libtensorflow_framework.so.1
#5  0x00007ff461c7a944 in _GLOBAL__sub_I_cpu_feature_guard.cc ()
   from /usr/local/google/home/zhuo/venv/en/local/lib/python2.7/site-packages/tensorflow/python/../libtensorflow_framework.so.1
#6  0x00007ff492a208aa in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffcc10b93a8,
    env=env@entry=0x56280ad136d0) at dl-init.c:72
#7  0x00007ff492a209bb in call_init (env=0x56280ad136d0, argv=0x7ffcc10b93a8, argc=1, l=<optimized out>) at dl-init.c:30
#8  _dl_init (main_map=main_map@entry=0x56280b2de470, argc=1, argv=0x7ffcc10b93a8, env=0x56280ad136d0) at dl-init.c:120
#9  0x00007ff492a24f38 in dl_open_worker (a=a@entry=0x7ffcc10b74c0) at dl-open.c:575
#10 0x00007ff492a20754 in _dl_catch_error (objname=objname@entry=0x7ffcc10b74b0, errstring=errstring@entry=0x7ffcc10b74b8,
    mallocedp=mallocedp@entry=0x7ffcc10b74af, operate=operate@entry=0x7ff492a24b50 <dl_open_worker>,
    args=args@entry=0x7ffcc10b74c0) at dl-error.c:187
#11 0x00007ff492a246e9 in _dl_open (          
    file=0x7ff48ee18c34 "/usr/local/google/home/zhuo/venv/en/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_
internal.so", mode=-2147483646, caller_dlopen=0x562809367582 <_PyImport_GetDynLoadFunc+226>, nsid=-2, argc=<optimized out>,
    argv=<optimized out>, env=0x56280ad136d0) at dl-open.c:660
#12 0x00007ff4925f0ee9 in dlopen_doit (a=a@entry=0x7ffcc10b76f0) at dlopen.c:66
#13 0x00007ff492a20754 in _dl_catch_error (objname=0x56280acc6e60, errstring=0x56280acc6e68, mallocedp=0x56280acc6e58,
    operate=0x7ff4925f0e90 <dlopen_doit>, args=0x7ffcc10b76f0) at dl-error.c:187
#14 0x00007ff4925f1531 in _dlerror_run (operate=operate@entry=0x7ff4925f0e90 <dlopen_doit>, args=args@entry=0x7ffcc10b76f0)
    at dlerror.c:163
#15 0x00007ff4925f0f82 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#16 0x0000562809367582 in _PyImport_GetDynLoadFunc ()
#17 0x00005628093671b4 in _PyImport_LoadDynamicModule ()
#18 0x00005628093f842f in ?? ()
#19 0x000056280930906a in PyEval_EvalFrameEx ()
#20 0x000056280930dd02 in PyEval_EvalFrameEx ()
#21 0x0000562809306865 in PyEval_EvalCodeEx ()
#22 0x0000562809306189 in PyEval_EvalCode ()
#23 0x00005628093060c5 in PyImport_ExecCodeModuleEx ()
#24 0x00005628092ff303 in ?? ()
#25 0x00005628092f13d7 in ?? ()
#26 0x00005628092f0dc4 in PyImport_ImportModuleLevel ()
#27 0x00005628092f23b8 in ?? ()
#28 0x00005628092f214e in PyObject_Call ()
#29 0x0000562809312580 in PyEval_CallObjectWithKeywords ()
#30 0x000056280930b2d4 in PyEval_EvalFrameEx ()
#31 0x0000562809306865 in PyEval_EvalCodeEx ()
#32 0x0000562809306189 in PyEval_EvalCode ()
#33 0x00005628093060c5 in PyImport_ExecCodeModuleEx ()
#34 0x00005628092ff303 in ?? ()
#35 0x00005628092f13d7 in ?? ()

[xhochy]

This is solved by the linked PRs. Please reopen if this persists after a 0.14 release.

[brills]

Hi Uwe,
did you mean this commit? 0289af2#diff-3c274588934dfc0526783b59d1f4fff2 ?

will there be a minor release soon?

[paulgc]

This seems to happen again after 0.14 release

Stack trace:

#0  0x0000000000000000 in ?? ()
#1  0x00007efc67801667 in __pthread_once_slow (
    once_control=0x7efc3b17db40 <assign_descriptors_table_google_2fprotobuf_2fdescriptor_2eproto>, init_routine=0x7efc3ff9dd60 <__once_proxy>)
    at pthread_once.c:116
#2  0x00007efc3ae7f56e in google::protobuf::internal::AssignDescriptors(google::protobuf::internal::AssignDescriptorsTable*) ()
   from /usr/local/google/home/pachristopher/venv/lib/python3.5/site-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so
#3  0x00007efc3ae49920 in google::protobuf::FieldDescriptorProto_Label_descriptor() ()
   from /usr/local/google/home/pachristopher/venv/lib/python3.5/site-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so
#4  0x00007efc3adfe9c2 in google::protobuf::python::InitDescriptor() ()
   from /usr/local/google/home/pachristopher/venv/lib/python3.5/site-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so
#5  0x00007efc3ade8130 in google::protobuf::python::InitProto2MessageModule(_object*) ()
   from /usr/local/google/home/pachristopher/venv/lib/python3.5/site-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so
#6  0x00007efc3ade54d2 in PyInit__message ()
   from /usr/local/google/home/pachristopher/venv/lib/python3.5/site-packages/google/protobuf/pyext/_message.cpython-35m-x86_64-linux-gnu.so
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x00005612fecab8a1 in _PyImport_LoadDynamicModuleWithSpec ()
#8  0x00005612fecabd50 in ?? ()
#9  0x00005612fec0cfb6 in PyCFunction_Call ()
#10 0x00005612febdc5eb in PyEval_EvalFrameEx ()
#11 0x00005612febde786 in ?? ()
#12 0x00005612febdae14 in PyEval_EvalFrameEx ()
#13 0x00005612febda28f in PyEval_EvalFrameEx ()
#14 0x00005612febda28f in PyEval_EvalFrameEx ()
#15 0x00005612febda28f in PyEval_EvalFrameEx ()
#16 0x00005612febda28f in PyEval_EvalFrameEx ()
#17 0x00005612febdf5fd in PyEval_EvalCodeEx ()
#18 0x00005612fec0edb3 in ?? ()
#19 0x00005612fec5dbb7 in PyObject_Call ()
#20 0x00005612fec5e862 in _PyObject_CallMethodIdObjArgs ()
#21 0x00005612febcf915 in PyImport_ImportModuleLevelObject ()
#22 0x00005612febe7e38 in ?? ()
#23 0x00005612fec0cf27 in PyCFunction_Call ()
#24 0x00005612febdc5eb in PyEval_EvalFrameEx ()
#25 0x00005612febde786 in ?? ()
#26 0x00005612febdae14 in PyEval_EvalFrameEx ()
#27 0x00005612febdf5fd in PyEval_EvalCodeEx ()
#28 0x00005612fec0edb3 in ?? ()
#29 0x00005612fec5dbb7 in PyObject_Call ()

[wesm]

We have no intention of fixing this right now. Please use conda if you need to use these projects together