segmentation fault with multi-threading

[lmiq]

I have this MWE, where I get segmentation faults (frequently, but not deterministically), when trying to run some script that uses multi-threading on the Julia side.

I have used before launching ipython3:

export JULIA_NUM_THREADS=4

(my computer has 4 cores - 8 threads).

The MWE is:

Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from juliacall import Main as jl

In [2]: import numpy as np

In [3]: jl.seval("""
   ...: function test(x)
   ...:     partial = zeros(Threads.nthreads())
   ...:     Threads.@threads for i in 1:Threads.nthreads()
   ...:         for j in i:Threads.nthreads():length(x)
   ...:             partial[i] += x[j]
   ...:         end
   ...:     end
   ...:     return sum(partial)
   ...: end
   ...: """)
Out[3]: test (generic function with 1 method)

In [4]: x = np.random.random((10_000,))

In [5]: %timeit jl.test(x)
72.2 µs ± 35.9 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: %timeit jl.test(x)
Segmentation fault (core dumped)

Here I have emulated the error using the %timeit macro from ipython, but my actual error I get after some runs of a function of my package:

In [1]: from juliacall import Main as jl

In [2]: jl.seval("using CellListMap")

In [3]: import numpy as np

In [5]: x = np.random.random((10_000,3))

In [6]: nb = jl.neighborlist(x.transpose(), 0.05)

In [7]: nb = jl.neighborlist(x.transpose(), 0.05)

In [8]: nb = jl.neighborlist(x.transpose(), 0.05)

In [9]: nb = jl.neighborlist(x.transpose(), 0.05)

In [10]: nb = jl.neighborlist(x.transpose(), 0.05)

In [11]: nb = jl.neighborlist(x.transpose(), 0.05)

In [12]: nb = jl.neighborlist(x.transpose(), 0.05)

In [13]: nb = jl.neighborlist(x.transpose(), 0.05)

In [14]: nb = jl.neighborlist(x.transpose(), 0.05)

In [15]: nb = jl.neighborlist(x.transpose(), 0.05)

In [16]: nb = jl.neighborlist(x.transpose(), 0.05)

In [17]: nb = jl.neighborlist(x.transpose(), 0.05)

In [18]: nb = jl.neighborlist(x.transpose(), 0.05)

In [19]: nb = jl.neighborlist(x.transpose(), 0.05)
Segmentation fault (core dumped)

%timeit runs the function multiple times, there seems to be some memory corruption, or memory overflow, causing the error.

Anyway, even if you have only some hint on how to debug this, I will be very thankful.

(even in the simplest example above, the segfaults only occur with multi-threading).

[cjdoris]

This is probably the same as #201 and #202.

The Python interpreter must only be used on thread 1. If you use Julia's multithreading, then even if it doesn't explicitly use Python, at some point the GC may free a Python object. If this GC runs on a thread other than 1, Python will crash.

The solution is to guard multithreaded Julia code with calls to PythonCall.GC.disable() and PythonCall.GC.enable(), so the Julia code in your MWE becomes:

function test(x)
    partial = zeros(Threads.nthreads())
    PythonCall.GC.disable()
    try
        Threads.@threads for i in 1:Threads.nthreads()
            for j in i:Threads.nthreads():length(x)
                partial[i] += x[j]
            end
        end
    finally
        PythonCall.GC.enable()
    end
    return sum(partial)
end

This ensures that no Python objects are freed during the multithreaded portion: if Julia's GC collects any Python objects, they will be cached and freed at PythonCall.GC.enable().

[lmiq]

Not sure if I did something wrong, but I still get the segfaults:

In [1]: from juliacall import Main as jl

In [2]: jl.seval("""
   ...: function test(x)
   ...:     partial = zeros(Threads.nthreads())
   ...:     PythonCall.GC.disable()
   ...:     try
   ...:         Threads.@threads for i in 1:Threads.nthreads()
   ...:             for j in i:Threads.nthreads():length(x)
   ...:                 partial[i] += x[j]
   ...:             end
   ...:         end
   ...:     finally
   ...:         PythonCall.GC.enable()
   ...:     end
   ...:     return sum(partial)
   ...: end
   ...: """)
Out[2]: test (generic function with 1 method)

In [3]: import numpy as np

In [4]: x = np.random.random((10_000,));

In [6]: %timeit jl.test(x)
60.4 µs ± 30.7 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [7]: %timeit jl.test(x)
Segmentation fault (core dumped)

(the same for my package)

[cjdoris]

In that case I don't know what's going on. I can reproduce the error but don't get a core dump. Can you poke around the core dump with gdb and see what's going on? Maybe get a backtrace?

[cjdoris]

If I do jl.GC.enable(False) before %timeit then the problem stops.

The PythonCall GC code is itself not thread safe, but I assumed that that wasn't an issue since Julia's GC can only run from one thread at a time, so I assumed that finalizers only ran in one thread at a time. Maybe that's not the case. I think with a little more digging I can figure this out.

[cjdoris]

I'm flummoxed!

Here's a slightly smaller MWE:

In [1]: import os

In [2]: os.environ['PYTHON_JULIACALL_THREADS'] = '4'

In [3]: from juliacall import Main as jl

In [4]: jl.seval("""
   ...: function test()
   ...:     partial = zeros(Threads.nthreads())
   ...:     x = zeros(10_000)
   ...:     Threads.@threads for i in 1:Threads.nthreads()
   ...:         for j in i:Threads.nthreads():length(x)
   ...:             partial[i] += x[j]
   ...:         end
   ...:     end
   ...:     return sum(partial)
   ...: end
   ...: """)
Out[4]: test (generic function with 1 method)

In [5]: %timeit jl.test()
The slowest run took 10.87 times longer than the fastest. This could mean that an intermediate result is being cached.
150 µs ± 103 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: %timeit jl.test()
Segmentation fault

Things tried:

• Deleting all the finalizers/GC stuff from PythonCall, so when a Python object is done with, it is never freed. Segfault still occurs.
• Running with 1 thread (either by setting the number of threads to 1 or removing @threads) fixes it.
• Calling jl.GC.enable(False) before %timeit ... fixes it.
• Doing %timeit jl.seval("test()") instead fixes it. Weird!

So it looks like some combination of garbage collection and threading are the issue. But I've turned off the finalizers in PythonCall so I don't know what else could be the issue.

[nmheim]

If I am using this approach to disable/enable GC I am running out of memory at some point. Would it help if I make a minimal example that reproduces this behaviour?

[cjdoris]

It's not surprising that disabling GC entirely means you run out of memory. You could maybe do a GC.collect() periodically (outside of any threaded code).

[nmheim]

Hmm, but I am enabling GC again after the threaded call. I cant find a GC.collect function. I tried GC.gc (outside threaded calls) but that results in a segfault. Thank you for your help!:)

[vchuravy]

As said on discourse: https://discourse.julialang.org/t/floop-threading-and-juliacall-produce-segmentation-fault/100717/11?u=vchuravy

Julia intentionally causes segmentation faults as part of the GC safepoint mechanism. This MPI.jl issue is somewhat related
JuliaParallel/MPI.jl#725

PythonCall starts Julia with --handle-signals=no

https://github.com/cjdoris/PythonCall.jl/blob/e374e2503b7b843c457d30596a2428ebfc26bb0b/pysrc/juliacall/__init__.py#L132

Which then causes benign segmentation faults to become spurios ones.

[giordano]

To be clear, you're suggesting to change that option to "yes" (or just remove it)? According to git blame, it was introduced last year in bef6afa without much explanation (just "better compatibility"), it's unclear what was the motivation for that change.

[vchuravy]

Yes, changing it from no to yes will at least no longer cause faults due to GC (and it looks like most of the recorded issues here worked around it by running with GC off.

This issue was opened shortly after that change aswell.

[giordano]

I see that the offending commit is part of the branch https://github.com/cjdoris/PythonCall.jl/commits/gc referenced in issues #201 and #202 which was meant to deal with other segmentation faults in the GC in multithreading programs, but it sounds like this specific change just worsened the situation. Is that an accurate reconstruction of the events @cjdoris?