Segfault with jl_init__threading with julia 1.6 on macOS

[Non-Contradiction]

On macOS, if we load the julia 1.6 dynamical library and segfault will happen at the jl_init__threading function. The same segfault doesn't happen with julia 1.5.

For example, with julia 1.6 in R

dyn.load("/Applications/Julia-1.6.app/Contents/Resources/julia/lib/libjulia.1.6.dylib")
.C("jl_init__threading")

which causes the following segfault:

*** caught segfault ***
address 0xfffffffffffffff8, cause 'memory not mapped'

Traceback:
1: .C("jl_init__threading")

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
ERROR: Selection:

And the same code runs fine with julia 1.5 in R

dyn.load("/Applications/Julia-1.5.app/Contents/Resources/julia/lib/libjulia.1.5.dylib")
.C("jl_init__threading")

With julia 1.6 in python3

MacBookdeMacBook-Pro:~ macbookpro$ python3
Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> libjulia = ctypes.PyDLL("/Applications/Julia-1.6.app/Contents/Resources/julia/lib/libjulia.1.6.dylib", ctypes.RTLD_GLOBAL)
>>> libjulia.jl_init__threading()
ERROR: Segmentation fault: 11
MacBookdeMacBook-Pro:~ macbookpro$ 

And the same code runs fine with julia 1.5 in python3

Python 3.8.2 (default, Dec 21 2020, 15:06:04) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> libjulia = ctypes.PyDLL("/Applications/Julia-1.5.app/Contents/Resources/julia/lib/libjulia.1.5.dylib", ctypes.RTLD_GLOBAL)    
>>> libjulia.jl_init__threading()
775
>>> exit()

Discourse post at https://discourse.julialang.org/t/segfault-when-jl-init-in-r-also-python-with-julia-1-6-on-macos/58110/3.
Relevant issue at JuliaInterop/JuliaCall#164.

[TheCedarPrince]

I can confirm this being an issue on Mac OS as I had the same error occur.

[JackDunnNZ]

@staticfloat Apologies for the direct ping, but do you know what might be going on here? It looks like jl_init__threading is segfaulting for 1.6 on macOS only.

[staticfloat]

Can you check with the latest master; I think I may have just fixed this as a part of ac7974a

[Non-Contradiction]

Thanks @staticfloat . I tried the binary from "https://s3.amazonaws.com/julialangnightlies/pretesting/mac/x64/1.7/julia-ac7974acef-mac64.dmg", the old error seems getting fixed, but new error appears which is still some missing symbol and segfault:

jl_flush_cstdio - dlsym(0x7fc548d88110, jl_flush_cstdio): symbol not found

[JackDunnNZ]

Thanks @staticfloat for the help!

I built from master and see the same as @Non-Contradiction with JuliaCall from R:

> JuliaCall::julia_setup()
Julia version 1.7.0-DEV.998 at location /private/tmp/julia/julia/usr/bin will be used.
Error in juliacall_initialize(.julia$dll_file, .julia$bin_dir, relative_sysimage_path) :
  jl_flush_cstdio - dlsym(0x7f809066f520, jl_flush_cstdio): symbol not found

which if it helps points to here:

https://github.com/Non-Contradiction/JuliaCall/blob/4777b771303bf7683a60f1cb75a88b5c3a53ecea/src/libjulia.cpp#L133

On Python/pyjulia, I see a similar error with a different symbol:

>>> from julia import Julia
>>> Julia(runtime='/tmp/julia/julia/julia')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/site-packages/julia/core.py", line 680, in __init__
    self.__julia = Julia(*args, **kwargs)
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/site-packages/julia/core.py", line 472, in __init__
    self.api = LibJulia.from_juliainfo(jlinfo)
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/site-packages/julia/libjulia.py", line 195, in from_juliainfo
    sysimage=juliainfo.sysimage,
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/site-packages/julia/libjulia.py", line 217, in __init__
    setup_libjulia(self.libjulia)
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/site-packages/julia/libjulia.py", line 34, in setup_libjulia
    libjulia.jl_.argtypes = [c_void_p]
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/Users/jack/.pyenv/versions/3.6.6/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7f8d8c4471c0, jl_): symbol not found

which points to here:

https://github.com/JuliaPy/pyjulia/blob/87c669e2729f9743fe2ab39320ec9b91c9300a96/src/julia/libjulia.py#L34

In both cases it looks like the error is hit before jl_init__xxx is called, so I can't say if the original problem is fixed

I also tried running the commands from @Non-Contradiction in the OP to load the image directly:

> dyn.load("/tmp/julia/julia/usr/lib/julia/sys.dylib")
Error in dyn.load("/tmp/julia/julia/usr/lib/julia/sys.dylib") :
  unable to load shared object '/tmp/julia/julia/usr/lib/julia/sys.dylib':
  dlopen(/tmp/julia/julia/usr/lib/julia/sys.dylib, 6): Library not loaded: @rpath/libjulia-internal.dylib
  Referenced from: /private/tmp/julia/julia/usr/lib/julia/sys.dylib
  Reason: image not found

>>> import ctypes
>>> libjulia = ctypes.PyDLL("/tmp/julia/julia/usr/lib/julia/sys.dylib", ctypes.RTLD_GLOBAL)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/tmp/julia/julia/usr/lib/julia/sys.dylib, 10): Library not loaded: @rpath/libjulia-internal.dylib
  Referenced from: /private/tmp/julia/julia/usr/lib/julia/sys.dylib
  Reason: image not found

I'm not sure if I am doing something wrong here though after the libjulia-internal change?

[staticfloat]

The jl_ undefined error is not because of the libjulia-internal change, it's because jl_ was un-exported in 67f29d8#diff-cf6e417d01a6e25b814df855f594119c5d96891b39091abc802a5b6b04a86a56L2

Is it possible to no longer use that symbol in pyjulia? If not, you may need to ask @vchuravy what to use instead.

You should not be loading sys.dylib directly and you shouldn't need to load libjulia-internal; you should only ever need to be loading libjulia.dylib.

[staticfloat]

It looks like the jl_flush_cstdio export was removed as well, which explains the other error.

[vtjnash]

It seems like neither package uses the symbol, they just check for the existence of it?

[JackDunnNZ]

Thanks both for the help! Unfortunately I don't know much about the internals of either package, but hopefully this helps @Non-Contradiction and @tkf

[Non-Contradiction]

@staticfloat @vtjnash Thanks for the help!
Yes, the jl_flush_cstdio symbol is no longer used. After removing the symbol checking for jl_flush_cstdio, JuliaCall does not complain about missing symbol anymore, but still segfault at jl_init__threading, which seems to be the same segfault if you load the libjulia dynamic library and call the function directly as in the original question.

[staticfloat]

Can you try invoking jl_init() instead of jl_init__threading()? It appears to me that this area is in flux a bit, as c556bb9 recently eliminated the jl_init__threading() export completely, so I assume that jl_init() is the "correct" entrypoint. Jameson, Jeff, feel free to contradict me if that's not the case.

[jacobxk]

same with this issue, and I have tried to reinstall Julia 1.6.1 and RCall, but still.

[fingolfin]

Note that c556bb9 broke ABI compatibility: any binary compiled against Julia <= 1.6 and calling jl_init() won't work with latest Julia master / Julia 1.7 unless it is recompiled.

If that's truly intentional, I guess we'll have to keep up producing multiple binaries for lots of BinaryBuilder JLLs, one for each major Julia version (and in some cases maybe more, as in the best there were ABI breaks even in patch level updates from 1.x.y to 1.x.(y+1)).

However, this migration is tricky, because there won't be a libjulia_jll 1.7 before Julia 1.7 is released, so I can't provide such JLLs linked against 1.7/master until then. This would be less of an issue if there was a clear strategy for such breaking changes in Julia: like, leaving a compatibility stub for jl_init__threading in 1.7 so that binaries compiled against older Julia version keep working, and removing that only in 1.8.