Limit initial OpenBLAS thread count

[staticfloat]

We set OpenBLAS's initial thread count to 1 to prevent runaway allocation within OpenBLAS's initial thread startup. LinearAlgebra will later call BLAS.set_num_threads() to the actual value we require.

[vtjnash]

Also GOTO_NUM_THREADS? And I think you can recover the test for this from
https://github.com/JuliaLang/julia/pull/42442/files#diff-e299fbf3f1935b1bb9e7ef61e4644dda078ae51fab7559caba8da03817943c9b

[staticfloat]

I think you can recover the test for this from
https://github.com/JuliaLang/julia/pull/42442/files#diff-e299fbf3f1935b1bb9e7ef61e4644dda078ae51fab7559caba8da03817943c9b

I don't see how that is testing this functionality; that's testing that we limit the number of threads to a maximum of 8, but that's not what we do anymore; and in fact, our logic is much closer to what OpenBLAS itself natively does, so it's difficult to test that this is working.

[vtjnash]

Ah, true. Now that we are later also explicitly changing it, we would need to do that check in __init__, before we set it.

[JeffBezanson]

LinearAlgebra.__init__ only calls set_num_threads if !haskey(ENV, "OPENBLAS_NUM_THREADS"); does this interfere with that?

[staticfloat]

No; so what happens is, if a thread count is already set, neither this logic nor the LinearAlgebra __init__() logic messes with it.

[vtjnash]

I think I agree with Jeff that this will break LinearAlgebra.__init__, since it will be set now

[staticfloat]

Oh, of course, my bad. I totally misunderstood the question, and confused myself.

One possibility is to unset the environment variable after dlopen() has returned, but I'm not crazy about that idea. What do you think, Jameson?

[vtjnash]

I would perhaps rather just patch openblas to default to 1 thread at startup, though this might harm other apps that use it

[staticfloat]

Okay, I'll just add a short-circuit to OpenBLAS's default thread calculation then. X-ref: JuliaPackaging/Yggdrasil#5555

[staticfloat]

I am attempting to upstream this patch here: OpenMathLib/OpenBLAS#3773

[PallHaraldsson]

I'm exited to see this merged, so I can test/use on master, since its only point is faster Julia startup, right? How faster expected?

So, is the "1 failing" check a false alarm? Does the test need to be rerun?

At the very start for freebsd64:

fcntl(): Bad file descriptor
Executing precompile statements... 124/1927
fcntl(): Bad file descriptor
[..]
Executing precompile statements... 682/1927
Killed
gmake[1]: *** [sysimage.mk:89: /usr/home/julia/buildbot/w1_builder/package_freebsd64/build/usr/lib/julia/sys-o.a] Error 1
Executing precompile statements... 683/1927
[..]

[staticfloat]

Sadly, I can confirm that while this does help a little, it's not very significant. Here are the "commit charge" graphs for the start of the testset for both master and this branch:

master

this branch

Isolation tests

With gdb, I have verified that the changes cause OpenBLAS to see only one thread at startup (which is subsequently increased when LinearAlgebra.__init__() sets the number of threads). Looking at the startup commit charge for master versus this branch, I do see a difference:

In the above screenshot, PID 5820 is this branch, whereas PID 2052 is master, and both are simply sitting at the REPL after startup with no other options applied. So we should be saving some amount of memory, but apparently it is insignificant compared to the memory that we are using during testing.

[staticfloat]

Alright, we finally managed to do some more testing on this on some large core-count Windows machines, and this helps significantly, so I'm going to merge.

[KristofferC]

Just for my understanding, for a large core-count machine this would at most save half of the memory compared to before? With this change, we set OpenBLAS to 1 thread, then LinearAlgebra.__init__ runs:

julia/stdlib/LinearAlgebra/src/LinearAlgebra.jl

Line 646 in 58b559f

BLAS.set_num_threads(max(1, Sys.CPU_THREADS ÷ 2))

and set the number of OpenBLAS threads to Sys.CPU_THREADS ÷ 2 (which is half the threads OpenBLAS would set earlier`)

If so, even with this, it is probably worth doing something like #46844 (setting the number of openblas threads to 1 for most of the test suite).

[DilumAluthge]

Yeah, ref JuliaCI/julia-buildkite#247 for changes specific to CI.

[staticfloat]

Actually, something seemed fishy to me about how much this helped the internal workload, and I just tried the following on Julia v1.8.2:

julia> using Distributed
       addprocs(1)
       @everywhere begin
           using LinearAlgebra
           @show LinearAlgebra.BLAS.get_num_threads()
       end

LinearAlgebra.BLAS.get_num_threads() = 16
      From worker 2:    LinearAlgebra.BLAS.get_num_threads() = 1

So it appears that Distributed already attempts to set the BLAS threads to 1. Looking into the source, I found:

• julia/stdlib/Distributed/src/process_messages.jl

  Line 336 in 626f3b2

  Base.disable_library_threading()

• julia/base/initdefs.jl

  Line 431 in 626f3b2

  f()

• julia/stdlib/LinearAlgebra/src/LinearAlgebra.jl

  Line 639 in 626f3b2

  Base.at_disable_library_threading(() -> BLAS.set_num_threads(1))

So what this means is that we actually already try to restrict OpenBLAS to 1 thread on CI, but because OpenBLAS used to start up and immediately set a number of threads, we would have the problem of initially starting up and consuming a bunch of memory, then never letting go of it.

So all that being said, I believe that with this PR merged, we've actually solved the root problem. We should watch it closely on CI, but I think we've only ever tried to exercise single-threaded BLAS on CI so far, so we should just continue to do that. :)

[KristofferC]

That has to run after LinearAlgebra.__init__, no? So how does that help with memory? OpenBLAS doesn't free buffers if you reduce the numer of threads, right? So LinearAlgebra will bump the threads then Distributed will reduce it, but at that point it is already too late?

I was actually about to open an issue that we need to set the env var for OpenBLAS when spawning distributed workers exactly due to the argumentation above.

Is there a way to query OpenBLAS how many buffer it currently has allocated?

[staticfloat]

Ah, you're right; with this PR we have moved from:

OpenBLAS DLL init (N threads) -> LinearAlgebra.__init__() (N/2 threads) -> disable_library_threading() (1 thread)

to now:

OpenBLAS DLL init (1 thread) -> LinearAlgebra.__init__() (N/2 threads) -> disable_library_threading() (1 thread)

So perhaps we need to change __init__() to not set those threads when we're a Distributed worker.

[KristofferC]

So perhaps we need to change init() to not set those threads when we're a Distributed worker.

Yes, or set env var when spawning workers unless OpenBLAS threading is explicitly enabled.

[staticfloat]

Yes, or set env var when spawning workers unless OpenBLAS threading is explicitly enabled.

I would be totally okay with that, but I'm not sure exactly where to put that, as I have a hard time disentangling the different cluster managers in Distributed, to find the local Distributed code paths, as opposed to the remote/SSH ones.

[KristofferC]

Is there anyway we could be more lazy and not set the number of BLAS threads explicitly until a BLAS operation is actually executed?

[staticfloat]

There is always the option of adding an if statement at the beginning of every BLAS operation, but I don't know how feasible that is.

[KristofferC]

Can it be done with libblastrampoline somehow? That intercepts every call going into BLAS so maybe the logic can be there?

[PallHaraldsson]

Thanks for merging!

on some large core-count Windows machines, and this helps significantly

It also helps on my (16-vthread) Linux taking 4.4 ms off (for down to 152.7 ms min. But best mean time is though 2.5 ms worse, likely because I got very lucky with the max value) and every ms counts for some benchmarking (where we're really close to other languages). I've gotten startup down by half with a non-default sysimage, for an older version, a much larger effect, but this too would help (yes, it's only 2.8% faster startup, with sysimage the gain should be amplified to about 5.6% faster).

$ hyperfine 'OPENBLAS_NUM_THREADS=1 julia-1.10-DEV-ee0f3fc334/bin/julia --startup-file=no --history-file=no --compile=min -O0 -e ""'
Benchmark 1: OPENBLAS_NUM_THREADS=1 julia-1.10-DEV-ee0f3fc334/bin/julia --startup-file=no --history-file=no --compile=min -O0 -e ""
  Time (mean ± σ):     181.9 ms ±  14.8 ms    [User: 110.3 ms, System: 71.7 ms]
  Range (min … max):   152.7 ms … 197.4 ms    15 runs

$ hyperfine 'julia-1.10-DEV-ee0f3fc334/bin/julia --startup-file=no --history-file=no --compile=min -O0 -e ""'
Benchmark 1: julia-1.10-DEV-ee0f3fc334/bin/julia --startup-file=no --history-file=no --compile=min -O0 -e ""
  Time (mean ± σ):     173.9 ms ±  14.0 ms    [User: 101.3 ms, System: 72.6 ms]
  Range (min … max):   157.1 ms … 194.3 ms    15 runs

I ran hyperfine many times, could never get min that far down without the ENV, while with it often higher than the min for without it.

I don't think I'm just measuring noise, some real effect, though the machine likely heats up while benchmarking. I tried to minimize all noise, turned off the web browser, but load didn't go down to 0.