Error when importing matplotlib and JuliaCall from Python

[dingraha]

I'm on Red Hat Enterprise Linux 8.6, running Python 3.9.7. I haven't installed Julia, but juliapkg appears to download and install Julia 1.8.4 just fine. I've created and activated a virtual environment, then installed matplotlib and juliacall:

$ virtualenv -p `which python3.9` venv
$ source ./venv/bin/activate
$ pip install matplotlib juliacall

In my working directory I have one Julia file:

$ cat Foo.jl
struct Foo end
$

and two Python files. The first Python file includes the Foo.jl file:

$ cat foo.py
import os

# Create a new Julia module that will hold all the Julia code imported into this Python module.
import juliacall; jl = juliacall.newmodule("FooModule")

# Get the directory this file is in, then include the `Foo.jl` Julia source code.
d = os.path.dirname(os.path.abspath(__file__))
jl.include(os.path.join(d, "Foo.jl"))

Foo = jl.Foo
$

And the second one imports the first foo.py file:

$ cat run.py
import matplotlib
from foo import Foo
$

When I run the run.py file, I get this error:

$ python run.py
ERROR: `ccall` requires the compilerTraceback (most recent call last):
  File "/home/dingraha/projects/pythoncall_import_error/run.py", line 2, in <module>
    from foo import Foo
  File "/home/dingraha/projects/pythoncall_import_error/foo.py", line 4, in <module>
    import juliacall; jl = juliacall.newmodule("FooModule")
  File "/home/dingraha/projects/pythoncall_import_error/venv/lib/python3.9/site-packages/juliacall/__init__.py", line 218, in <module>
    init()
  File "/home/dingraha/projects/pythoncall_import_error/venv/lib/python3.9/site-packages/juliacall/__init__.py", line 214, in init
    raise Exception('PythonCall.jl did not start properly')
Exception: PythonCall.jl did not start properly
$

Now, if I switch the order of the two import lines in run.py, there's no error. Also if I import juliacall first, i.e.:

$ cat run-no_error.py
import juliacall
import matplotlib
from foo import Foo
$

the error goes away.

Any ideas?

[dingraha]

Actually, an even simpler MWE is to just import matplotlib and then juliacall from one file:

$ cat foo2.py
import matplotlib
import juliacall
$ python foo2.py
ERROR: `ccall` requires the compilerTraceback (most recent call last):
  File "/home/dingraha/projects/pythoncall_import_error/foo2.py", line 2, in <module>
    import juliacall
  File "/home/dingraha/projects/pythoncall_import_error/venv/lib/python3.9/site-packages/juliacall/__init__.py", line 218, in <module>
    init()
  File "/home/dingraha/projects/pythoncall_import_error/venv/lib/python3.9/site-packages/juliacall/__init__.py", line 214, in init
    raise Exception('PythonCall.jl did not start properly')
Exception: PythonCall.jl did not start properly
$

Here is what I have installed in the virtual environment:

$ pip list
Package          Version
---------------- -------
contourpy        1.0.6
cycler           0.11.0
fonttools        4.38.0
juliacall        0.9.10
juliapkg         0.1.9
kiwisolver       1.4.4
matplotlib       3.6.2
numpy            1.24.1
packaging        22.0
Pillow           9.4.0
pip              22.3.1
pyparsing        3.0.9
python-dateutil  2.8.2
semantic-version 2.10.0
setuptools       65.6.3
six              1.16.0
wheel            0.38.4
$

[cjdoris]

Thanks for the detailed report. That's a very odd error message that I don't understand at all, and Googling for "Julia compiler traceback" returns nothing. Looks like a Julia internal thing to investigate.

Julia 1.8.5 is released, you could upgrade (juliapkg.resolve(force=True) I think) to see if that happens to fix it.

[dingraha]

Yep, Julia 1.8.5 does seem to fix the error. I'm going to try to test this a bit more, then I'll close the issue. Thanks for the help!

[dingraha]

OK, I spoke too soon. I'm testing this with two approaches to using Python:

• Using the RHEL system Python with virtualenv
• Using mamba/conda after installing mambaforge from here

I've tried Julia 1.8.4 and 1.8.5. If I run this script:

import matplotlib
import juliacall

with a Python from a conda environment and either Julia 1.8.4 or 1.8.5, everything is fine.
I've tried this with conda environments using Python 3.9.7 and 3.11.0.
But using the system Python on RHEL (Python 3.9.7) in a virtual enviroment gives the error reported previously, with both Julia 1.8.4 and 1.8.5.
So the bug seems to be related to the way Python is installed.

I know that most people discourage using the system Python for code development, so maybe this is a non-issue?

[cjdoris]

Hmm. Well the error arises in this line of the Julia source code: https://github.com/JuliaLang/julia/blob/ba69cbaa8e04657b6f4dceced76696575053f28b/src/interpreter.c#L338

As the error message says, it's because the Julia compiler is not available for some reason - which seems weird. I don't know what can cause that to occur.

My best guess is that your system Python has been built in a non-standard way that prevents Julia from linking to its compiler. Maybe it's statically linked? Maybe it's linked to an old version of LLVM? I can only guess. What's the output of ldd python (where python is your system Python)?

As you say, the system Python is typically only there to support the system. You're generally expected to install Python yourself if you actually want to use it.

[dingraha]

Here's the output of ldd when I use a virtual environment with the system Python:

$ which python
~/desk/pythoncall_wtf/venv-mybuild-with-libc-enable-shared-without-lto-without-optimizations-computed-gotos-no-dtrace-no-ssl/bin/python
$ ldd `which python`
        linux-vdso.so.1 (0x00007fff0decd000)
        libpython3.9.so.1.0 => /lib64/libpython3.9.so.1.0 (0x00007f62e5949000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f62e5720000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f62e5500000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f62e52fc000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f62e50f8000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f62e4d76000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f62e49b1000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f62e44c8000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f62e5f15000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f62e42b0000)
$

And this is what I get for ldd using the Python conda installs:

$ which python
~/local/mambaforge/envs/juliacall_wtf_python3.9.7/bin/python
$ ldd `which python`
        linux-vdso.so.1 (0x00007fffc557a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f372494b000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f3724747000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f3724543000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f372433b000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f3723fb9000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f3723bf4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3724f5a000)
$

I don't see any significant differences there.

A few other data points: I don't see this bug when running tests on GitHub Actions Ubuntu machines using this in the YAML file:

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}

I also don't run into this bug when using the system Python on Arch Linux.

Another hint: If I set JULIA_DEBUG=all and then try, I see this:

$ cat test.py
import matplotlib
import juliacall
$ JULIA_DEBUG=all python test.py
┌ Debug: Loading cache file /home/dingraha/.julia/compiled/v1.8/CompilerSupportLibraries_jll/iCwSB_gBZDr.ji for CompilerSupportLibraries_jll [e66e0078-7015-5450-92f7-15fbd957f2ae]
└ @ Base loading.jl:806
ERROR: `ccall` requires the compilerTraceback (most recent call last):
  File "/home/dingraha/desk/pythoncall_wtf/test.py", line 2, in <module>
    import juliacall
  File "/home/dingraha/desk/pythoncall_wtf/venv-mybuild-with-libc-enable-shared-without-lto-without-optimizations-computed-gotos-no-dtrace-no-ssl/lib/python3.9/site-packages/juliacall/__init__.py", line 218, in <module>
    init()
  File "/home/dingraha/desk/pythoncall_wtf/venv-mybuild-with-libc-enable-shared-without-lto-without-optimizations-computed-gotos-no-dtrace-no-ssl/lib/python3.9/site-packages/juliacall/__init__.py", line 214, in init
    raise Exception('PythonCall.jl did not start properly')
Exception: PythonCall.jl did not start properly
$

So the problem appears to have something to do with CompilerSupportLibraries_jll, which apparently provides things like libgomp and libstdcxx.

I wonder if the problem is related to how old the C/C++-related stuff is on Red Hat? On Red Hat 8.6, I have gcc-8.5.0 and libc-2.28. On the Arch Linux system I tried, I have gcc-12.2.1 and libc-2.36. My current theory is that importing matplotlib on RHEL first loads an older version of some C library that matplotlib needs which isn't compatible with the recent version of Julia that juliacall installs. But importing juliacall first loads a more-recent version of the library from CompilerSupportLibraries_jll that is backwards-compatible with matplotlib. And we don't see the issue with a Python installed with Conda because it installs a more-recent version of the mysterious library.

Not sure how to debug this further. Any ideas?

[cjdoris]

Is it possible to inspect which libraries have been linked after matplotlib is loaded?

Your theory sounds totally plausible - if you use PythonCall from Julia, we ensure the Conda environment is compatible with the version of libstdc++ which Julia has installed, otherwise nasty errors occur. Maybe something similar happens using JuliaCall from Python - e.g. matplotlib could be loading a too-old version of libstdc++.

[dingraha]

Is it possible to inspect which libraries have been linked after matplotlib is loaded?

Oh, good idea!

matplotlib could be loading a too-old version of libstdc++

Yes, that appears to be the problem! Turns out you can see what shared libraries are being loaded by setting LD_DEBUG=libs in the environment. So, if I have python scripts called pass.py and fail.py

(venv) $ cat pass.py
import juliacall
import matplotlib
print("hi")
(venv) $ cat fail.py
import matplotlib
import juliacall
print("hi")
(venv) $

I can run pass.py without any error:

(venv) $ LD_DEBUG=libs python pass.py 2> pass.stderr
hi
(venv) $

Then if I look through pass.stderr for references for libstdc++, I'll find this:

     15700:     find library=libstdc++.so.6 [0]; searching
     15700:      search cache=/etc/ld.so.cache
     15700:       trying file=/lib64/libstdc++.so.6
     15700:
     15700:     find library=libgcc_s.so.1 [0]; searching
     15700:      search cache=/etc/ld.so.cache
     15700:       trying file=/lib64/libgcc_s.so.1
     15700:
     15700:
     15700:     calling init: /lib64/libgcc_s.so.1
     15700:
     15700:
     15700:     calling init: /lib64/libstdc++.so.6
     15700:
     15700:     /lib64/libstdc++.so.6: error: symbol lookup error: undefined symbol: GLIBCXX_3.4.31 (fatal)
     15683:     find library=libstdc++.so.6 [0]; searching
     15683:      search cache=/etc/ld.so.cache
     15683:       trying file=/lib64/libstdc++.so.6
     15683:
     15683:     find library=libgcc_s.so.1 [0]; searching
     15683:      search path=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/glibc-hwcaps/x86-64-v3:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/glibc-hwcaps/x86-64-v2:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/haswell/x86_64:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/haswell:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/x86_64:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/haswell/x86_64:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/haswell:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/x86_64:/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia            (RUNPATH from file /home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/libstdc++.so.6)
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/glibc-hwcaps/x86-64-v3/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/glibc-hwcaps/x86-64-v2/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/haswell/x86_64/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/haswell/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/x86_64/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/tls/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/haswell/x86_64/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/haswell/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/x86_64/libgcc_s.so.1
     15683:       trying file=/home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/libgcc_s.so.1
     15683:
     15683:
     15683:     calling init: /home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/libgcc_s.so.1
     15683:
     15683:
     15683:     calling init: /home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/libstdc++.so.6
     15683:

Looks like the script tries to load the system libstdc++ at /lib64/libstdc++.so.6, can't, then eventually uses Julia's libstdc++ (which is compatible with what matplotlib needs).

But for fail.py:

(venv) $ LD_DEBUG=libs python fail.py 2> fail.stderr
(venv) $

The script fails, and I see the usual error in fail.stderr.
Now looking for references to libstdc++ in fail.stderr, I find stuff like this:

     15813:     find library=libstdc++.so.6 [0]; searching
     15813:      search cache=/etc/ld.so.cache
     15813:       trying file=/lib64/libstdc++.so.6
     15813:
     15813:
     15813:     calling init: /lib64/libstdc++.so.6
     15813:
     15813:

And then later:

     15813:     /lib64/libstdc++.so.6: error: version lookup error: version `GLIBCXX_3.4.26' not found (required by /home/dingraha/desk/pythoncall_wtf/venv/julia_env/pyjuliapkg/install/lib/julia/libjulia-codegen.so.1) (fatal)

So, mystery solved. Looks like a mention of this would fit in well with the FAQs in the docs. I will submit a PR.

Thanks!

[cjdoris]

Closing since I think the only fix is to document the issue, which is done now (thank you).