Description
This issue is somewhat complex and slightly above my understanding level to explain correctly, so instead I am copy-pasting the Slack thread where I described the problem and where the actual issue was diagnosed and a workaround was eventually discovered (so that the whole thing doesn't disappear in the Slack hole). I am putting this in a collapsible item in order not to make this issue too long, but I advise you to read the whole thread here or on Slack to get a full idea of the issue.
Slack thread
Christophe Grandsire-Koevoets
Yesterday at 14:30
Hi all, just updated to 1.8.0 on my Ubuntu 18.04 VM, but when I try to run it I get the following:
$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org/
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.0 (2022-08-17)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
`ccall` requires the compiler
SYSTEM: caught exception of type
And then Julia simply hangs. 1.7.3 runs fine. Any idea what's going on?
77 replies
Valentin (NOT vchuravy)
🧽 20 hours ago
what do you have in your startup.jl?
Harmen Stoppels
20 hours ago
Does it go away when you temporarily remove/rename julia's libstdc++.so? That's how I've seen the issue during a build of julia (edited)
Christophe Grandsire-Koevoets
20 hours ago
I only have code to start OhMyREPL and Revise if they're found in the environment. But it doesn't matter. If I delete startup.jl, I get:
$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org/
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.0 (2022-08-17)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
SYSTEM: caught exception of type
And Julia hangs again.
Christophe Grandsire-Koevoets
20 hours ago
@harmen_stoppels
Let me see if I can located that file...
Valentin (NOT vchuravy)
🧽 20 hours ago
how did you install?
Christophe Grandsire-Koevoets
20 hours ago
I found libstdc++.so (and libstd+++.so.6 and libstd++.so.6.0.29) but no combination of removing those files or keeping them makes Julia start
Christophe Grandsire-Koevoets
20 hours ago
I installed via asdf, but all it does is pull the official binaries.
Christophe Grandsire-Koevoets
20 hours ago
Tried uninstalling and reinstalling, to no avail.
Valentin (NOT vchuravy)
🧽 20 hours ago
hmm works for me with official binaries
Valentin (NOT vchuravy)
🧽 20 hours ago
as in, downloaded from the website
Christophe Grandsire-Koevoets
20 hours ago
asdf gets the binary from https://julialang-s3.julialang.org/bin/linux/x64/1.8/julia-1.8.0-linux-x86_64.tar.gz
, which as far as I can tell is the same link as on the website.
Valentin (NOT vchuravy)
🧽 20 hours ago
indeed it is
Christophe Grandsire-Koevoets
20 hours ago
To test that asdf doesn't break anything, I tried downloading the tar.gz from the official website, untar it in a folder and running Julia from there, but I get the same problem. So it really comes from the official binary.
Christophe Grandsire-Koevoets
20 hours ago
Once again, no other Julia version I have installed shows this problem...
Can it have to do with Ubuntu 18.04? It's an old version but still supported, so I wouldn't expect a problem with that (been meaning to upgrade my VM but never got any time to do so).
Valentin (NOT vchuravy)
🧽 20 hours ago
your first error ("ccall requires the compiler") suggests to me that maybe some linking step is going wrong on your machine?
Valentin (NOT vchuravy)
🧽 20 hours ago
though I think 1.7 was the first to seperate codegen from the runtime, so that shouldn't happen with 1.8..
Christophe Grandsire-Koevoets
20 hours ago
But why would that hit 1.8.0 but not 1.7.3?
Valentin (NOT vchuravy)
🧽 20 hours ago
no idea 🤷
Harmen Stoppels
20 hours ago
if you run it with env -i /path/to/julia-1.8.0/bin/julia, does that work?
Christophe Grandsire-Koevoets
20 hours ago
$ env -i ~/.asdf/installs/julia/1.8.0/bin/julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org/
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.8.0 (2022-08-17)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
┌ Warning: Terminal not fully functional
└ @ Base client.jl:410
julia>
So it kinda works.
Harmen Stoppels
20 hours ago
it's probably LD_LIBRARY_PATH
, does it work with LD_LIBRARY_PATH= ./julia
(edited)
Christophe Grandsire-Koevoets
20 hours ago
Yes!
🎉
1
Harmen Stoppels
19 hours ago
pointing to another julia lib dir? 😜
Christophe Grandsire-Koevoets
19 hours ago
Unfortunately, I need LD_LIBRARY_PATH
set up for other programs I'm running. And no, it doesn't seem to point to anything Julia-like:
$ echo $LD_LIBRARY_PATH
/usr/lib/x86_64-linux-gnu:/home/christophe/.oracle/instantclient_19_8::/usr/local/MATLAB/MATLAB_Runtime/v96/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/extern/bin/glnxa64
Christophe Grandsire-Koevoets
19 hours ago
Let me try something...
Christophe Grandsire-Koevoets
19 hours ago
OK, I cannot start Julia unless my LD_LIBRARY_PATH
is empty. Which is just not feasible for me. Why does Julia suddenly mess up with this? Until 1.7.3 I had no problem...
Valentin (NOT vchuravy)
🧽 19 hours ago
not sure if that's related, but there's a double : in your LD_LIBRARY_PATH
Valentin (NOT vchuravy)
🧽 19 hours ago
could it be that there's some other library in one of those paths that is overriding one of the libraries julia ships?
Christophe Grandsire-Koevoets
19 hours ago
I removed the double ::, but it's indeed unrelated. And yes, that's probably the issue. Unfortunately, I have no choice but to have LD_LIBRARY_PATH
set up that way, or other programs I need won't work. And once again, never had an issue until 1.8.0, so I don't get why it's suddenly a problem.
Harmen Stoppels
19 hours ago
still thinking it's a libstdc++.so shadowing the julia or system version (edited)
☝️
1
Valentin (NOT vchuravy)
🧽 19 hours ago
is there anything else printed after the SYSTEM: caught exception of type?
Christophe Grandsire-Koevoets
19 hours ago
Agreed. But once again, no choice here. I have to have that environment variable set up the way it is, and Julia has had no problem with it until now.
Christophe Grandsire-Koevoets
19 hours ago
@Sukera
No, Julia hangs at that point
Valentin (NOT vchuravy)
🧽 19 hours ago
do you have the TERM environment variable set?
Valentin (NOT vchuravy)
🧽 19 hours ago
what other julia environment variables have you set?
Christophe Grandsire-Koevoets
19 hours ago
$ echo $TERM
xterm-256color
Harmen Stoppels
19 hours ago
i think i see the issue
Valentin (NOT vchuravy)
🧽 19 hours ago
that doesn't make sense
Valentin (NOT vchuravy)
🧽 19 hours ago
you only get the "terminal not fully functional" error if TERM isn't set and you're not on windows
Valentin (NOT vchuravy)
🧽 19 hours ago
term_env = get(ENV, "TERM", @static Sys.iswindows() ? "" : "dumb")
term = REPL.Terminals.TTYTerminal(term_env, stdin, stdout, stderr)
banner && Base.banner(term)
if term.term_type == "dumb"
repl = REPL.BasicREPL(term)
quiet || @warn "Terminal not fully functional"
Harmen Stoppels
19 hours ago
libjulia-codegen.so.1.8
sets an rpath, others are using runpath, that's usually a recipe for disaster
Harmen Stoppels
19 hours ago
what does ldd libjulia-codegen.so.1.8
give you? (with your dirty environment) (edited)
Christophe Grandsire-Koevoets
19 hours ago
$ ldd libjulia-codegen.so.1.8
ldd: ./libjulia-codegen.so.1.8: No such file or directory
Valentin (NOT vchuravy)
🧽 19 hours ago
at the place of the library, so in usr/lib of your julia install
Christophe Grandsire-Koevoets
19 hours ago
Oops, sorry. It's kinda above my level already. Gimme a second
😆
1
Christophe Grandsire-Koevoets
19 hours ago
$ ldd libjulia-codegen.so.1.8
linux-vdso.so.1 (0x00007ffe653b2000)
libunwind.so.8 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libunwind.so.8 (0x00007ffb61825000)
libLLVM-13jl.so => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libLLVM-13jl.so (0x00007ffb5d16d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffb5cf69000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffb5cd61000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffb5cb42000)
libatomic.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libatomic.so.1 (0x00007ffb61aee000)
libjulia.so.1 => not found
libjulia-internal.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libjulia-internal.so.1 (0x00007ffb5c581000)
libstdc++.so.6 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libstdc++.so.6 (0x00007ffb5c36d000)
libgcc_s.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libgcc_s.so.1 (0x00007ffb61ad3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffb5bf7c000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffb61a58000)
libz.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libz.so.1 (0x00007ffb5bd61000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffb5b9c3000)
libjulia.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./../libjulia.so.1 (0x00007ffb61aaf000)
Harmen Stoppels
19 hours ago
yeah, libjulia.so.1 => not found worries me a little bit, it's thanks to rpath+runpath mixing.
Valentin (NOT vchuravy)
🧽 19 hours ago
it's found though?
Valentin (NOT vchuravy)
🧽 19 hours ago
just in the next line
Valentin (NOT vchuravy)
🧽 19 hours ago
wait
Valentin (NOT vchuravy)
🧽 19 hours ago
what
Valentin (NOT vchuravy)
🧽 19 hours ago
the entry is doubled
Valentin (NOT vchuravy)
🧽 19 hours ago
check the last line
Harmen Stoppels
19 hours ago
ah, it's a direct dep and a transitive dep, as a direct dep it's not located cause the rpath is wrong (doesn't include $ORIGIN/..), but it's found down the line (edited)
Harmen Stoppels
19 hours ago
does this change anything for you?
patchelf --set-rpath '$ORIGIN/:$ORIGIN/../' libjulia-codegen.so.1.8
Christophe Grandsire-Koevoets
19 hours ago
Give me a moment, I have a call right now.
Christophe Grandsire-Koevoets
18 hours ago
OK, did what you asked
@harmen_stoppels
, and now I still can't run Julia without changing the LD_LIBRARY_PATH, but I also get this:
christophe@Ubuntu-VirtualBox: ~/.asdf/installs/julia/1.8.0/lib/julia ((v0.10.2))
$ patchelf --set-rpath '$ORIGIN/:$ORIGIN/../' libjulia-codegen.so.1.8
christophe@Ubuntu-VirtualBox: ~/.asdf/installs/julia/1.8.0/lib/julia ((v0.10.2))
$ ldd libjulia-codegen.so.1.8
./libjulia-codegen.so.1.8: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./libjulia-codegen.so.1.8)
linux-vdso.so.1 (0x00007ffd9df77000)
libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f5af8f1c000)
libLLVM-13jl.so => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libLLVM-13jl.so (0x00007f5af4864000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5af4660000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5af4458000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5af4239000)
libatomic.so.1 => /usr/lib/x86_64-linux-gnu/libatomic.so.1 (0x00007f5af4031000)
libjulia.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./../libjulia.so.1 (0x00007f5af91a4000)
libjulia-internal.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libjulia-internal.so.1 (0x00007f5af3a70000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5af36e7000)
libgcc_s.so.1 => /usr/local/MATLAB/MATLAB_Runtime/v96/sys/os/glnxa64/libgcc_s.so.1 (0x00007f5af34d0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5af30df000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5af9137000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5af2eb9000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5af2c9c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5af28fe000)
Christophe Grandsire-Koevoets
18 hours ago
LD_LIBRARY_PATH
seems to mess everything up indeed.
Harmen Stoppels
18 hours ago
at least my suspicion about libstdc++ was right 😜 yeah, so matlab ships another set of compiler suport libraries...
you could try to replace the runpath elsewhere with rpath too so that it does not listen to LD_LIBRARY_PATH
Harmen Stoppels
18 hours ago
can you run libtree libjulia-codegen.so.1.8
it may tell you where LD_LIBRARY_PATH is actually overriding things.
Harmen Stoppels
18 hours ago
https://github.com/haampie/libtree
Christophe Grandsire-Koevoets
18 hours ago
$ ~/Downloads/libtree libjulia-codegen.so.1.8
libjulia-codegen.so.1
├── libunwind.so.8 [LD_LIBRARY_PATH]
│ └── liblzma.so.5 [ld.so.conf]
│ └── libpthread.so.0 [ld.so.conf]
├── libatomic.so.1 [LD_LIBRARY_PATH]
│ └── libpthread.so.0 [ld.so.conf]
├── libLLVM-13jl.so [runpath]
│ ├── librt.so.1 [ld.so.conf]
│ │ └── libpthread.so.0 [ld.so.conf]
│ ├── libpthread.so.0 [ld.so.conf]
│ └── libz.so.1 [ld.so.conf]
├── libjulia-internal.so.1 [runpath]
│ ├── libunwind.so.8 [LD_LIBRARY_PATH]
│ ├── libatomic.so.1 [LD_LIBRARY_PATH]
│ ├── libz.so.1 [runpath]
│ ├── libjulia.so.1 [runpath]
│ │ └── libpthread.so.0 [ld.so.conf]
│ ├── librt.so.1 [ld.so.conf]
│ └── libpthread.so.0 [ld.so.conf]
├── libjulia.so.1 [runpath]
├── librt.so.1 [ld.so.conf]
└── libpthread.so.0 [ld.so.conf]
Harmen Stoppels
18 hours ago
if you do: patchelf --force-rpath libjulia-codegen.so.1
and patchelf --force-rpath libjulia-internal.so.1
does that fix it
Christophe Grandsire-Koevoets
18 hours ago
Nope 😞
Harmen Stoppels
18 hours ago
okay lol, patchelf is a broken piece of software. you have to use it like patchelf --force-rpath --set-rpath '$ORIGIN/:$ORIGIN/../' [lib]
(edited)
Christophe Grandsire-Koevoets
18 hours ago
Success!
Christophe Grandsire-Koevoets
18 hours ago
$ ~/Downloads/libtree libjulia-codegen.so.1.8
libjulia-codegen.so.1
├── libunwind.so.8 [rpath]
│ └── libz.so.1 [runpath]
├── libLLVM-13jl.so [rpath]
│ ├── libz.so.1 [rpath of 1]
│ ├── librt.so.1 [ld.so.conf]
│ │ └── libpthread.so.0 [ld.so.conf]
│ └── libpthread.so.0 [ld.so.conf]
├── libatomic.so.1 [rpath]
│ └── libpthread.so.0 [ld.so.conf]
├── libjulia-internal.so.1 [rpath]
│ ├── libunwind.so.8 [rpath]
│ ├── libz.so.1 [rpath]
│ ├── libatomic.so.1 [rpath]
│ ├── libjulia.so.1 [rpath]
│ │ └── libpthread.so.0 [ld.so.conf]
│ ├── librt.so.1 [ld.so.conf]
│ └── libpthread.so.0 [ld.so.conf]
├── libjulia.so.1 [rpath]
├── librt.so.1 [ld.so.conf]
└── libpthread.so.0 [ld.so.conf]
Christophe Grandsire-Koevoets
18 hours ago
And julia finally runs without workarounds
Christophe Grandsire-Koevoets
18 hours ago
But what a mess...
Harmen Stoppels
18 hours ago
lol, it's still rather questionable... in particular that libz is detected through the rpath of libjulia-codegen... 😬
Harmen Stoppels
18 hours ago
to be fair: LD_LIBRARY_PATH containing a non-system version of gcc runtime libs is asking for broken executables...
☝️
1
Harmen Stoppels
18 hours ago
however, julia's libs are not in the best state either.
Christophe Grandsire-Koevoets
18 hours ago
Not looking forward to do that at each Julia update...
Valentin (NOT vchuravy)
🧽 18 hours ago
opening an issue for that sounds reasonable
Valentin (NOT vchuravy)
🧽 18 hours ago
no matter that MATLAB seemingly intentionally breaks things here, julia should continue to work
Harmen Stoppels
18 hours ago
just open an issue, maybe julia should use rpaths, but for sure it should fix the incomplete r(un)path of libjulia-codegen (which depends on libjulia but is not in its search path) (edited)
💯
1
Christophe Grandsire-Koevoets
18 hours ago
Mmm... I need to take a shower, walk the dog and have dinner. I'll definitely open an issue, but that'll have to wait until tomorrow. I'll update this thread once that's done.
Thanks for the help!
As for the TL;DR, and once again, it's slightly above my geek level, so I may be summarising this wrong (which is why I added the full thread in here):
- I installed the newest Julia 1.8.0 on my Ubuntu 18.04 VM using
asdf
, which pulls the official Julia binaries; - Trying to run the REPL failed, even with no base environment and no
startup.jl
file present. The REPL preamble would appear and then Julia would printSYSTEM: caught exception of type
and then it would hang. Nothing short of killing the Julia process would allow me to access the terminal again; - After a lot of help and back and forth, it was discovered that I had
LD_LIBRARY_PATH
set, so that I could run some other stuff on my computer (including the MATLAB Runtime). Unfortunately,libjulia-codegen.so.1.8
seemed to be set to listening toLD_LIBRARY_PATH
when looking for its dependencies, and those found the wrong libraries, including trying to uselibstdc++.so
from the MATLAB Runtime, which being the wrong version caused everything to fail; - By running
patchelf --force-rpath
on bothlibjulia-codegen.so.1.8
andlibjulia-internal.so.1.8
, I was able to get the right dependencies to be found, and was able to run Julia 1.8.0 without a problem from then on.
Now, from what I understand, what MATLAB did (having a set of non-standard gcc runtime libs in LD_LIBRARY_PATH
) is a recipe for disasters, but even then Julia's libs are not in the best state either, what with libjulia-codegen.so.1.8
having an incomplete r(un)path, and it was worth opening an issue to solve this problem.
So I'm opening this issue. Once again, sorry if this issue is not of the best quality, but how Linux shared libraries work is not my specialty, and I know nothing of rpath or runpath. So no idea if this is easy to solve or not. I just know that I have no choice in the matter of having LD_LIBRARY_PATH
set the way it is, and Julia shouldn't hang just because other programs make use of it, even if their use of it is questionable at best.
BTW, here's the output of versioninfo()
now that I can run Julia 1.8.0 without a problem:
Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 8 × Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
Threads: 1 on 8 virtual cores
Environment:
LD_LIBRARY_PATH = /usr/lib/x86_64-linux-gnu:/home/christophe/.oracle/instantclient_19_8:/usr/local/MATLAB/MATLAB_Runtime/v96/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/extern/bin/glnxa64
Thanks for your understanding.