Skip to content

Julia 1.8.0 hangs on startup when LD_LIBRARY_PATH is set #46409

Closed
@tsela

Description

@tsela

This issue is somewhat complex and slightly above my understanding level to explain correctly, so instead I am copy-pasting the Slack thread where I described the problem and where the actual issue was diagnosed and a workaround was eventually discovered (so that the whole thing doesn't disappear in the Slack hole). I am putting this in a collapsible item in order not to make this issue too long, but I advise you to read the whole thread here or on Slack to get a full idea of the issue.

Slack thread

Christophe Grandsire-Koevoets
Yesterday at 14:30
Hi all, just updated to 1.8.0 on my Ubuntu 18.04 VM, but when I try to run it I get the following:

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org/
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0 (2022-08-17)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

`ccall` requires the compiler

SYSTEM: caught exception of type

And then Julia simply hangs. 1.7.3 runs fine. Any idea what's going on?
77 replies

Valentin (NOT vchuravy)
🧽 20 hours ago
what do you have in your startup.jl?

Harmen Stoppels
20 hours ago
Does it go away when you temporarily remove/rename julia's libstdc++.so? That's how I've seen the issue during a build of julia (edited)

Christophe Grandsire-Koevoets
20 hours ago
I only have code to start OhMyREPL and Revise if they're found in the environment. But it doesn't matter. If I delete startup.jl, I get:

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org/
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0 (2022-08-17)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |


SYSTEM: caught exception of type 

And Julia hangs again.

Christophe Grandsire-Koevoets
20 hours ago
@harmen_stoppels
Let me see if I can located that file...

Valentin (NOT vchuravy)
🧽 20 hours ago
how did you install?

Christophe Grandsire-Koevoets
20 hours ago
I found libstdc++.so (and libstd+++.so.6 and libstd++.so.6.0.29) but no combination of removing those files or keeping them makes Julia start

Christophe Grandsire-Koevoets
20 hours ago
I installed via asdf, but all it does is pull the official binaries.

Christophe Grandsire-Koevoets
20 hours ago
Tried uninstalling and reinstalling, to no avail.

Valentin (NOT vchuravy)
🧽 20 hours ago
hmm works for me with official binaries

Valentin (NOT vchuravy)
🧽 20 hours ago
as in, downloaded from the website

Christophe Grandsire-Koevoets
20 hours ago
asdf gets the binary from https://julialang-s3.julialang.org/bin/linux/x64/1.8/julia-1.8.0-linux-x86_64.tar.gz, which as far as I can tell is the same link as on the website.

Valentin (NOT vchuravy)
🧽 20 hours ago
indeed it is

Christophe Grandsire-Koevoets
20 hours ago
To test that asdf doesn't break anything, I tried downloading the tar.gz from the official website, untar it in a folder and running Julia from there, but I get the same problem. So it really comes from the official binary.

Christophe Grandsire-Koevoets
20 hours ago
Once again, no other Julia version I have installed shows this problem...
Can it have to do with Ubuntu 18.04? It's an old version but still supported, so I wouldn't expect a problem with that (been meaning to upgrade my VM but never got any time to do so).

Valentin (NOT vchuravy)
🧽 20 hours ago
your first error ("ccall requires the compiler") suggests to me that maybe some linking step is going wrong on your machine?

Valentin (NOT vchuravy)
🧽 20 hours ago
though I think 1.7 was the first to seperate codegen from the runtime, so that shouldn't happen with 1.8..

Christophe Grandsire-Koevoets
20 hours ago
But why would that hit 1.8.0 but not 1.7.3?

Valentin (NOT vchuravy)
🧽 20 hours ago
no idea 🤷

Harmen Stoppels
20 hours ago
if you run it with env -i /path/to/julia-1.8.0/bin/julia, does that work?

Christophe Grandsire-Koevoets
20 hours ago

$ env -i ~/.asdf/installs/julia/1.8.0/bin/julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org/
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.0 (2022-08-17)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

┌ Warning: Terminal not fully functional
└ @ Base client.jl:410
julia> 

So it kinda works.

Harmen Stoppels
20 hours ago
it's probably LD_LIBRARY_PATH, does it work with LD_LIBRARY_PATH= ./julia (edited)

Christophe Grandsire-Koevoets
20 hours ago
Yes!
🎉
1

Harmen Stoppels
19 hours ago
pointing to another julia lib dir? 😜

Christophe Grandsire-Koevoets
19 hours ago
Unfortunately, I need LD_LIBRARY_PATH set up for other programs I'm running. And no, it doesn't seem to point to anything Julia-like:

$ echo $LD_LIBRARY_PATH 
/usr/lib/x86_64-linux-gnu:/home/christophe/.oracle/instantclient_19_8::/usr/local/MATLAB/MATLAB_Runtime/v96/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/extern/bin/glnxa64

Christophe Grandsire-Koevoets
19 hours ago
Let me try something...

Christophe Grandsire-Koevoets
19 hours ago
OK, I cannot start Julia unless my LD_LIBRARY_PATH is empty. Which is just not feasible for me. Why does Julia suddenly mess up with this? Until 1.7.3 I had no problem...

Valentin (NOT vchuravy)
🧽 19 hours ago
not sure if that's related, but there's a double : in your LD_LIBRARY_PATH

Valentin (NOT vchuravy)
🧽 19 hours ago
could it be that there's some other library in one of those paths that is overriding one of the libraries julia ships?

Christophe Grandsire-Koevoets
19 hours ago
I removed the double ::, but it's indeed unrelated. And yes, that's probably the issue. Unfortunately, I have no choice but to have LD_LIBRARY_PATH set up that way, or other programs I need won't work. And once again, never had an issue until 1.8.0, so I don't get why it's suddenly a problem.

Harmen Stoppels
19 hours ago
still thinking it's a libstdc++.so shadowing the julia or system version (edited)
☝️
1

Valentin (NOT vchuravy)
🧽 19 hours ago
is there anything else printed after the SYSTEM: caught exception of type?

Christophe Grandsire-Koevoets
19 hours ago
Agreed. But once again, no choice here. I have to have that environment variable set up the way it is, and Julia has had no problem with it until now.

Christophe Grandsire-Koevoets
19 hours ago
@Sukera
No, Julia hangs at that point

Valentin (NOT vchuravy)
🧽 19 hours ago
do you have the TERM environment variable set?

Valentin (NOT vchuravy)
🧽 19 hours ago
what other julia environment variables have you set?

Christophe Grandsire-Koevoets
19 hours ago

$ echo $TERM
xterm-256color

Harmen Stoppels
19 hours ago
i think i see the issue

Valentin (NOT vchuravy)
🧽 19 hours ago
that doesn't make sense

Valentin (NOT vchuravy)
🧽 19 hours ago
you only get the "terminal not fully functional" error if TERM isn't set and you're not on windows

Valentin (NOT vchuravy)
🧽 19 hours ago

            term_env = get(ENV, "TERM", @static Sys.iswindows() ? "" : "dumb")
            term = REPL.Terminals.TTYTerminal(term_env, stdin, stdout, stderr)
            banner && Base.banner(term)
            if term.term_type == "dumb"
                repl = REPL.BasicREPL(term)
                quiet || @warn "Terminal not fully functional"

Harmen Stoppels
19 hours ago
libjulia-codegen.so.1.8 sets an rpath, others are using runpath, that's usually a recipe for disaster

Harmen Stoppels
19 hours ago
what does ldd libjulia-codegen.so.1.8 give you? (with your dirty environment) (edited)

Christophe Grandsire-Koevoets
19 hours ago

$ ldd libjulia-codegen.so.1.8
ldd: ./libjulia-codegen.so.1.8: No such file or directory

Valentin (NOT vchuravy)
🧽 19 hours ago
at the place of the library, so in usr/lib of your julia install

Christophe Grandsire-Koevoets
19 hours ago
Oops, sorry. It's kinda above my level already. Gimme a second
😆
1

Christophe Grandsire-Koevoets
19 hours ago

$ ldd libjulia-codegen.so.1.8
	linux-vdso.so.1 (0x00007ffe653b2000)
	libunwind.so.8 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libunwind.so.8 (0x00007ffb61825000)
	libLLVM-13jl.so => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libLLVM-13jl.so (0x00007ffb5d16d000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffb5cf69000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffb5cd61000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffb5cb42000)
	libatomic.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libatomic.so.1 (0x00007ffb61aee000)
	libjulia.so.1 => not found
	libjulia-internal.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libjulia-internal.so.1 (0x00007ffb5c581000)
	libstdc++.so.6 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libstdc++.so.6 (0x00007ffb5c36d000)
	libgcc_s.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libgcc_s.so.1 (0x00007ffb61ad3000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffb5bf7c000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffb61a58000)
	libz.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libz.so.1 (0x00007ffb5bd61000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffb5b9c3000)
	libjulia.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./../libjulia.so.1 (0x00007ffb61aaf000)

Harmen Stoppels
19 hours ago
yeah, libjulia.so.1 => not found worries me a little bit, it's thanks to rpath+runpath mixing.

Valentin (NOT vchuravy)
🧽 19 hours ago
it's found though?

Valentin (NOT vchuravy)
🧽 19 hours ago
just in the next line

Valentin (NOT vchuravy)
🧽 19 hours ago
wait

Valentin (NOT vchuravy)
🧽 19 hours ago
what

Valentin (NOT vchuravy)
🧽 19 hours ago
the entry is doubled

Valentin (NOT vchuravy)
🧽 19 hours ago
check the last line

Harmen Stoppels
19 hours ago
ah, it's a direct dep and a transitive dep, as a direct dep it's not located cause the rpath is wrong (doesn't include $ORIGIN/..), but it's found down the line (edited)

Harmen Stoppels
19 hours ago
does this change anything for you?
patchelf --set-rpath '$ORIGIN/:$ORIGIN/../' libjulia-codegen.so.1.8

Christophe Grandsire-Koevoets
19 hours ago
Give me a moment, I have a call right now.

Christophe Grandsire-Koevoets
18 hours ago
OK, did what you asked
@harmen_stoppels
, and now I still can't run Julia without changing the LD_LIBRARY_PATH, but I also get this:

christophe@Ubuntu-VirtualBox: ~/.asdf/installs/julia/1.8.0/lib/julia ((v0.10.2))
$ patchelf --set-rpath '$ORIGIN/:$ORIGIN/../' libjulia-codegen.so.1.8
christophe@Ubuntu-VirtualBox: ~/.asdf/installs/julia/1.8.0/lib/julia ((v0.10.2))
$ ldd libjulia-codegen.so.1.8
./libjulia-codegen.so.1.8: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./libjulia-codegen.so.1.8)
	linux-vdso.so.1 (0x00007ffd9df77000)
	libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f5af8f1c000)
	libLLVM-13jl.so => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libLLVM-13jl.so (0x00007f5af4864000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5af4660000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5af4458000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5af4239000)
	libatomic.so.1 => /usr/lib/x86_64-linux-gnu/libatomic.so.1 (0x00007f5af4031000)
	libjulia.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./../libjulia.so.1 (0x00007f5af91a4000)
	libjulia-internal.so.1 => /home/christophe/.asdf/installs/julia/1.8.0/lib/julia/./libjulia-internal.so.1 (0x00007f5af3a70000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5af36e7000)
	libgcc_s.so.1 => /usr/local/MATLAB/MATLAB_Runtime/v96/sys/os/glnxa64/libgcc_s.so.1 (0x00007f5af34d0000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5af30df000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5af9137000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5af2eb9000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5af2c9c000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5af28fe000)

Christophe Grandsire-Koevoets
18 hours ago
LD_LIBRARY_PATH seems to mess everything up indeed.

Harmen Stoppels
18 hours ago
at least my suspicion about libstdc++ was right 😜 yeah, so matlab ships another set of compiler suport libraries...
you could try to replace the runpath elsewhere with rpath too so that it does not listen to LD_LIBRARY_PATH

Harmen Stoppels
18 hours ago
can you run libtree libjulia-codegen.so.1.8 it may tell you where LD_LIBRARY_PATH is actually overriding things.

Harmen Stoppels
18 hours ago
https://github.com/haampie/libtree

Christophe Grandsire-Koevoets
18 hours ago

$ ~/Downloads/libtree libjulia-codegen.so.1.8 
libjulia-codegen.so.1 
├── libunwind.so.8 [LD_LIBRARY_PATH]
│   └── liblzma.so.5 [ld.so.conf]
│       └── libpthread.so.0 [ld.so.conf]
├── libatomic.so.1 [LD_LIBRARY_PATH]
│   └── libpthread.so.0 [ld.so.conf]
├── libLLVM-13jl.so [runpath]
│   ├── librt.so.1 [ld.so.conf]
│   │   └── libpthread.so.0 [ld.so.conf]
│   ├── libpthread.so.0 [ld.so.conf]
│   └── libz.so.1 [ld.so.conf]
├── libjulia-internal.so.1 [runpath]
│   ├── libunwind.so.8 [LD_LIBRARY_PATH]
│   ├── libatomic.so.1 [LD_LIBRARY_PATH]
│   ├── libz.so.1 [runpath]
│   ├── libjulia.so.1 [runpath]
│   │   └── libpthread.so.0 [ld.so.conf]
│   ├── librt.so.1 [ld.so.conf]
│   └── libpthread.so.0 [ld.so.conf]
├── libjulia.so.1 [runpath]
├── librt.so.1 [ld.so.conf]
└── libpthread.so.0 [ld.so.conf]

Harmen Stoppels
18 hours ago
if you do: patchelf --force-rpath libjulia-codegen.so.1 and patchelf --force-rpath libjulia-internal.so.1 does that fix it

Christophe Grandsire-Koevoets
18 hours ago
Nope 😞

Harmen Stoppels
18 hours ago
okay lol, patchelf is a broken piece of software. you have to use it like patchelf --force-rpath --set-rpath '$ORIGIN/:$ORIGIN/../' [lib] (edited)

Christophe Grandsire-Koevoets
18 hours ago
Success!

Christophe Grandsire-Koevoets
18 hours ago

$ ~/Downloads/libtree libjulia-codegen.so.1.8 
libjulia-codegen.so.1 
├── libunwind.so.8 [rpath]
│   └── libz.so.1 [runpath]
├── libLLVM-13jl.so [rpath]
│   ├── libz.so.1 [rpath of 1]
│   ├── librt.so.1 [ld.so.conf]
│   │   └── libpthread.so.0 [ld.so.conf]
│   └── libpthread.so.0 [ld.so.conf]
├── libatomic.so.1 [rpath]
│   └── libpthread.so.0 [ld.so.conf]
├── libjulia-internal.so.1 [rpath]
│   ├── libunwind.so.8 [rpath]
│   ├── libz.so.1 [rpath]
│   ├── libatomic.so.1 [rpath]
│   ├── libjulia.so.1 [rpath]
│   │   └── libpthread.so.0 [ld.so.conf]
│   ├── librt.so.1 [ld.so.conf]
│   └── libpthread.so.0 [ld.so.conf]
├── libjulia.so.1 [rpath]
├── librt.so.1 [ld.so.conf]
└── libpthread.so.0 [ld.so.conf]

Christophe Grandsire-Koevoets
18 hours ago
And julia finally runs without workarounds

Christophe Grandsire-Koevoets
18 hours ago
But what a mess...

Harmen Stoppels
18 hours ago
lol, it's still rather questionable... in particular that libz is detected through the rpath of libjulia-codegen... 😬

Harmen Stoppels
18 hours ago
to be fair: LD_LIBRARY_PATH containing a non-system version of gcc runtime libs is asking for broken executables...
☝️
1

Harmen Stoppels
18 hours ago
however, julia's libs are not in the best state either.

Christophe Grandsire-Koevoets
18 hours ago
Not looking forward to do that at each Julia update...

Valentin (NOT vchuravy)
🧽 18 hours ago
opening an issue for that sounds reasonable

Valentin (NOT vchuravy)
🧽 18 hours ago
no matter that MATLAB seemingly intentionally breaks things here, julia should continue to work

Harmen Stoppels
18 hours ago
just open an issue, maybe julia should use rpaths, but for sure it should fix the incomplete r(un)path of libjulia-codegen (which depends on libjulia but is not in its search path) (edited)
💯
1

Christophe Grandsire-Koevoets
18 hours ago
Mmm... I need to take a shower, walk the dog and have dinner. I'll definitely open an issue, but that'll have to wait until tomorrow. I'll update this thread once that's done.
Thanks for the help!

As for the TL;DR, and once again, it's slightly above my geek level, so I may be summarising this wrong (which is why I added the full thread in here):

  1. I installed the newest Julia 1.8.0 on my Ubuntu 18.04 VM using asdf, which pulls the official Julia binaries;
  2. Trying to run the REPL failed, even with no base environment and no startup.jl file present. The REPL preamble would appear and then Julia would print SYSTEM: caught exception of type and then it would hang. Nothing short of killing the Julia process would allow me to access the terminal again;
  3. After a lot of help and back and forth, it was discovered that I had LD_LIBRARY_PATH set, so that I could run some other stuff on my computer (including the MATLAB Runtime). Unfortunately, libjulia-codegen.so.1.8 seemed to be set to listening to LD_LIBRARY_PATH when looking for its dependencies, and those found the wrong libraries, including trying to use libstdc++.so from the MATLAB Runtime, which being the wrong version caused everything to fail;
  4. By running patchelf --force-rpath on both libjulia-codegen.so.1.8 and libjulia-internal.so.1.8, I was able to get the right dependencies to be found, and was able to run Julia 1.8.0 without a problem from then on.

Now, from what I understand, what MATLAB did (having a set of non-standard gcc runtime libs in LD_LIBRARY_PATH) is a recipe for disasters, but even then Julia's libs are not in the best state either, what with libjulia-codegen.so.1.8 having an incomplete r(un)path, and it was worth opening an issue to solve this problem.

So I'm opening this issue. Once again, sorry if this issue is not of the best quality, but how Linux shared libraries work is not my specialty, and I know nothing of rpath or runpath. So no idea if this is easy to solve or not. I just know that I have no choice in the matter of having LD_LIBRARY_PATH set the way it is, and Julia shouldn't hang just because other programs make use of it, even if their use of it is questionable at best.

BTW, here's the output of versioninfo() now that I can run Julia 1.8.0 without a problem:

Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores
Environment:
  LD_LIBRARY_PATH = /usr/lib/x86_64-linux-gnu:/home/christophe/.oracle/instantclient_19_8:/usr/local/MATLAB/MATLAB_Runtime/v96/runtime/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/bin/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/sys/os/glnxa64:/usr/local/MATLAB/MATLAB_Runtime/v96/extern/bin/glnxa64

Thanks for your understanding.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugIndicates an unexpected problem or unintended behaviorbuildingBuild system, or building Julia or its dependenciesregressionRegression in behavior compared to a previous version

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions