Skip to content

Commit 7186b8b

Browse files
JamesWrigleyDilumAluthge
authored andcommitted
Use Base.require() to load Distributed internally (#54571)
TL;DR: we changed to loading stdlibs exclusively from the default sysimg in #53326, and this breaks developing Distributed.jl because the workers will load the builtin version instead of the development version. I think this should be backported to 1.11? CC @jpsamaroo --- `Base.require_stdlib()` will exclusively load a stdlib from whatever was shipped with Julia. The problem is that when developing Distributed.jl this will cause the workers to always load the builtin Distributed module instead of the development version, which can break everything because now the master and worker are running different (potentially incompatible) versions of Distributed. This commit changes Base and LinearAlgebra to use `Base.require()` instead, which will respect `Base.LOAD_PATH`. I argue that this is safe because unlike the other stdlibs like REPL or Pkg, Distributed is only loaded if explicitly requested by the user with `-p` or through `addprocs()` or something, so it shouldn't be possible to get into quite the same tools-are-broken-because-I-broke-my-tools situation that motivated using `Base.require_stdlib()` in the first place. An alternative design would be to: 1. Move the if-block loading Distributed in `exec_options()` below the for-loop where it will execute `-e` options. 2. Require any implementation of `Distributed.launch(::ClusterManager)` to pass `-e 'using Distributed'` in their command to ensure that Distributed is loaded in a way respecting `Base.LOAD_PATH`. This would be more consistent with how the other stdlibs must be developed, but it requires implementers (i.e. Distributed and ClusterManagers.jl) to opt-in to allowing development versions of Distributed, which feels very annoying and easy to miss so I decided not to implement that.
1 parent 89e231c commit 7186b8b

File tree

2 files changed

+2
-2
lines changed

2 files changed

+2
-2
lines changed

base/client.jl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,7 +260,7 @@ function exec_options(opts)
260260
# Load Distributed module only if any of the Distributed options have been specified.
261261
distributed_mode = (opts.worker == 1) || (opts.nprocs > 0) || (opts.machine_file != C_NULL)
262262
if distributed_mode
263-
let Distributed = require_stdlib(PkgId(UUID((0x8ba89e20_285c_5b6f, 0x9357_94700520ee1b)), "Distributed"))
263+
let Distributed = require(PkgId(UUID((0x8ba89e20_285c_5b6f, 0x9357_94700520ee1b)), "Distributed"))
264264
Core.eval(MainInclude, :(const Distributed = $Distributed))
265265
Core.eval(Main, :(using Base.MainInclude.Distributed))
266266
end

stdlib/LinearAlgebra/src/LinearAlgebra.jl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -743,7 +743,7 @@ function peakflops(n::Integer=4096; eltype::DataType=Float64, ntrials::Integer=3
743743
end
744744

745745
if parallel
746-
let Distributed = Base.require_stdlib(Base.PkgId(
746+
let Distributed = Base.require(Base.PkgId(
747747
Base.UUID((0x8ba89e20_285c_5b6f, 0x9357_94700520ee1b)), "Distributed"))
748748
nworkers = @invokelatest Distributed.nworkers()
749749
results = @invokelatest Distributed.pmap(peakflops, fill(n, nworkers))

0 commit comments

Comments
 (0)