Parallelization on CPU segfaults on occasion

[amcostant]

I tried implementing a Palatalized diff() see code below

function pdiff(f::Array{T,1},x::T) where T <:SymEngine.Basic
       dfdx = Array{Basic}(undef,length(f))
       Threads.@threads for i = 1:length(f)
          dfdx[i]=diff(f[i],x)
       end
       return dfdx
end

julia> a,b = SymEngine.symbols("a b")
julia> ex = [rand(1)[1]*a^2+rand(1)[1]*a*b for i in 1:10000]
julia> pdiff(ex,a)

The code runs successfully part of the time and part of the time I get this error:

signal (11): Segmentation fault
in expression starting at no file:0

signal (11): Segmentation fault
in expression starting at no file:0

signal (11): Segmentation fault

signal (11): Segmentation fault

signal (11): Segmentation fault

signal (11): Segmentation fault
in expression starting at no file:0
in expression starting at no file:0
in expression starting at no file:0
in expression starting at no fileunknown function (ip: 0x7420e5b57749acff)
Allocaression starting at no file:0
Allocations: 11560826 (Pool: 11558345; Big: 2481); GC: 25
n expression starting at no file:0
in expression starting at no file:0
in expression starting at no file:0
in expression starting at no filein expression starting at no file:0
in expression starting at no filein expression starting at no file:0
in expression starting at no fileSegmentation fault (core dumped)`

and sometimes this:
'julia> pdiff(ex,a)

signal (11): Segmentation fault
in expression starting at no file:0

signal (11): Segmentation fault

signal (11): Segmentation fault
in expres11): Segmentation fault
in expression starting at no fileunknown function (ip: 0x1ecf6644a00fin expression starting at no filein expression starting at no file:0
n expression starting at no file:0
0ff)
in expression starting at no file:0
Allocations: 11779635 (Pool: 1177712Allocations: 11779635 (Pool: 1177712Allocations: 11779635 (Pool: 11777124; Big: 2511); GC: 25
Segmentation fault (core dumped)'

for info on my system

julia> versioninfo()
Julia Version 1.1.0
Commit 80516ca202 (2019-01-21 21:24 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, haswell)
Environment:
  JULIA_NUM_THREADS = 20

I have noticed that with more cores the problem is more persistent. For example when attempting to benchmark on two cores, I get this:

julia> @benchmark pdiff(ex,a)

Error thrown in threaded loop on thread 0: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 1: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 0: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 1: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 0: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 1: ReadOnlyMemoryError()
...
Error thrown in threaded loop on thread 1: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 0: ReadOnlyMemoryError()
Error thrown in threaded loop on thread 1: ReadOnlyMemoryError()corrupted double-linked list

Error thrown in threaded loop on thread 1: ReadOnlyMemoryError()
signal (6): Aborted
in expression starting at no file:0

The problem persists whenever I use more than one core, and I do not have this problem when running other parallel code.

[isuruf]

https://github.com/symengine/SymEngineBuilder/blob/master/build_tarballs.jl doesn't build the code with thread safety on. We should build with thread safety on.

[amcostant]

Hi Isuru, how would I go about modifying the "build_tarballs.jl" to include that? Thanks!

[amcostant]

Just an aside, parallelization on GPU works with CuArrays.jl

[isuruf]

@amcostant, add -DWITH_SYMENGINE_THREAD_SAFE=yes to https://github.com/symengine/SymEngineBuilder/blob/22960a8cb4750ebefa8163da9a839040a1a1d719/build_tarballs.jl#L20

[amcostant]

@isuruf I am sorry, but I am unsure how would I go about doing that, I can't seem to find "build_tarballs.jl" in the SymEngine julia package

[amcostant]

Thank you for fixing the issue, but the performance of multithreading is very poor, would this be an issue with SymEngine, my code, or Threads?

function pdiff(f::Array{T,1},x::T) where T <:SymEngine.Basic
       dfdx = Array{Basic}(undef,length(f))
       Threads.@threads for i = 1:length(f)
          @inbounds dfdx[i]=diff(f[i],x)
       end
       return dfdx
end

with 1 thread

julia> @benchmark pdiff(ex,a)
BenchmarkTools.Trial: 
  memory estimate:  1.75 MiB
  allocs estimate:  170003
  --------------
  minimum time:     34.431 ms (0.00% GC)
  median time:      35.414 ms (0.00% GC)
  mean time:        39.710 ms (2.58% GC)
  maximum time:     123.717 ms (43.19% GC)
  --------------
  samples:          126
  evals/sample:     1

with 4 threads

julia> @benchmark pdiff(ex,a)
BenchmarkTools.Trial: 
  memory estimate:  1.61 MiB
  allocs estimate:  163922
  --------------
  minimum time:     36.513 ms (0.00% GC)
  median time:      71.566 ms (0.00% GC)
  mean time:        65.060 ms (0.00% GC)
  maximum time:     79.225 ms (0.00% GC)
  --------------
  samples:          77
  evals/sample:     1

with 20 threads

julia> @benchmark pdiff(ex,a)
BenchmarkTools.Trial: 
  memory estimate:  1.62 MiB
  allocs estimate:  163964
  --------------
  minimum time:     73.878 ms (0.00% GC)
  median time:      85.815 ms (0.00% GC)
  mean time:        84.764 ms (0.00% GC)
  maximum time:     96.551 ms (0.00% GC)
  --------------
  samples:          59
  evals/sample:     1

Additionally, I attempted

function pdiff(b1::Array{Basic,1}, b2::Basic)
    dfdx = Array{Basic}(undef,length(b1))
    Threads.@threads for i = 1:length(b1)
        a = Basic()
        ret = ccall((:basic_diff, libsymengine), Int, (Ref{Basic}, Ref{Basic}, Ref{Basic}), a, b1[i], b2)
        dfdx[i] = a
    end
    return dfdx
end

and resulted in nearly identical performance.
I also checked that the system monitor shows the correct number of threads performing computations.

[amcostant]

@isuruf Would you happen to know why it is performing slower on multiple threads, do you have any advice to improve the performance, thanks.

[isuruf]

I tried locally yesterday on a 6 core machine. With 6 threads it was about half the time as in serial. One issue I could think of is that a, b are shared and ref counting with multiple threads has overhead due to locking.

[amcostant]

I tried to use @threadcall but ended up with this problem JuliaLang/julia#30864
Are there any solutions that would alleviate the problems you listed, thanks?

[amcostant]

@isuruf do you know if /build_tarballs.jl is built with -DWITH_OPENMP:BOOL=ON, I believe that might be the reason for the slow down if it is not the case.

[isuruf]

@amcostant, no, they are not. See https://github.com/symengine/SymEngineBuilder/blob/master/build_tarballs.jl#L20 for the options that we use

[amcostant]

@isuruf do you believe this could improve multi-core performance by adding that option?