@everywhere is slow on HPC with multi-node environment

[algorithmx]

julia/stdlib/Distributed/src/macros.jl

Line 207 in 7647ab5

remotecall_eval(Main, procs, ex)

Please check here for descriptions of the problem by three Julia users:

https://discourse.julialang.org/t/everywhere-takes-a-very-long-time-when-using-a-cluster/35724

I have tested @everywhere and pmap() on an HPC. Test code and result available here
https://github.com/algorithmx/nodeba

Basically I just put timestamps between the lines. You can see in t*.log files that the largest gap is the one between timestamp 3 and 4. More interestingly, I found that increasing nworkers() causes the gap to increase linearly. I believe that this gap represents the execution time of the macro @everywhere, seen from master.

The vesion info is :

julia> versioninfo()
Julia Version 1.5.3
Commit 788b2c7 (2020-11-09 13:37 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
CPU: AMD EPYC 7452 32-Core Processor
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, znver2)
Environment:
JULIA_PKG_SERVER = https://mirrors.tuna.tsinghua.edu.cn/julia

[algorithmx]

related issue #28966
@KristofferC

[arnauqb]

Hey @algorithmx I'm facing exactly the same issue you describe, have you found any workaround yet?

Could it be realted to this? JuliaLang/Pkg.jl#1219

[algorithmx]

Hey @algorithmx I'm facing exactly the same issue you describe, have you found any workaround yet?

Could it be realted to this? JuliaLang/Pkg.jl#1219

not yet :-(

[arnauqb]

So I have been able to reduce the delay quite significantly by using the latest Julia 1.6 release (seems that the faster compilation speed helps), and also changing the Base.DEPOT_PATH.

using Distributed, ClusterManagers
pids = addprocs_slurm(...)
@everywhere pushfirst!(Base.DEPOT_PATH, "/tmp/julia.cache")

[moble]

I've also run into this problem (posted on discourse here), and traced it back to just using @everywhere with basically any simple statement — even

@everywhere 1+2

It so happens that most of us run @everywhere using <SomePackage> first, so it looks like it has to do with precompilation, but I don't think it does. If I literally run @everywhere 1+2 first, then do all my imports, the imports are nice and fast — but only after 1+2 finishes, which takes forever.

This is a real killer for my use case, which involves scaling up to thousands of processors, which will waste (and has wasted for me) thousands of cpu-hours just running that first @everywhere statement.

[moble]

Also note that @affans reported that this was a regression, with julia 1.0.5 running very quickly, and 1.3 very slowly.

[KristofferC]

If possible, it would be good to run a bisect between julia 1.0 and 1.3 to find out if there was a specific commit that caused the regression.

[LarkAnspach]

Yes

[vancleve]

I've also noticed this on Julia 1.6.2 and it's not just multi-node environments. When I am on a 128 core AMD machine and perform @everywhere using pkgs, I notice using top that quickly only a handful of julia processes are using any CPU at once (~10% or so) and only one of the processes is running. Which process is running changes until finally the @everywhere using completes. This happen on multi-node systems too except its one node at a time with a handful of processes at low CPU and one process running.

I have a video of what this looks like on a single node here:
https://youtu.be/mTar7HvIMQo

[moble]

Workaround described here. Basically, you have to precompile the code that lets processes talk to each other.

[vancleve]

thanks @moble! I guess what I wonder is why is seems like precompilation problem is worse with many more processors? In other words, why aren't they all just precompiling simultaneously (the video makes it look like they're doing it one by one almost)?

[moble]

I don't actually know how things work under the hood, but I have tested it and found that the timing increases linearly with the number of processes. So my mental model is that @everywhere involves the primary process sending the instruction to workers and waiting for some type of confirmation that the instruction was received — or at least that sending has started. (I don't know it's an actual receipt confirmation, or opening of the socket, or creation of that worker's log, or the beginning of deserialization...)

But the primary must do this in serial to some extent, meaning it doesn't start sending the instruction to the next worker until it has whatever confirmation it needs. Normally this wouldn't be a problem, because confirmation is presumably almost instantaneous the great majority of the time. But compiling the code required to confirm takes ~1 second. And that's the part that has to happen on each worker in serial. That 1 second is not used in compiling the statement itself (which I know because I've tried statements that take much longer to compile); it must be just some piece of code required to let the primary know it got the message.

By precompiling something as simple as @everywhere 1+1, each worker can skip that step of compiling the confirming function, so the primary can move on to the next worker more quickly. And that's exactly what KristofferC has added/re-enabled in #42156.

[vancleve]

maybe the precompilation in #42156 didn't fix this issue?

#42156 (comment)

maybe there is some other code that's needs to be precompiled on the worker end that isn't precompiled by just calling @everywhere 1+1 or the other lines in generate_precompile.jl?

[moble]

Maybe. I don't know how julia's own build process works; maybe the image you used isn't being built with multiple processes.

Also, I'll point out that in the workaround I linked above, I actually used the --trace-compile flag on both the primary julia process and the worker process, then combined the outputs in case the worker's output wasn't a subset of the primary's. (I didn't actually check whether it was or not.) I don't know whether or not julia does this when building itself.