Short-cut parsing the compiler if you dev the loaded version

This adds a special mechanism to shortcut parsing/loading the whole compiler
if the version you have dev'ed happens to exactly match (under the usual criteria
for cache validation) match the version that you have compiled into the system
image. As written, this does not extend to `add`'ed packages. I would like to
make that work in the future, but the refactoring required is too extensive
for my current taste and somewhat independent. Fortuantely, it's just an
optimization, so it can be added later. Still, I would like to keep this
path for `dev` so that we can exercise it to see if there's any other problems with it.

Demo: With `dev stdlib/Compiler`, we get a small `7k` .so file, since the Compiler does not
get re-loaded.

```
julia> @time using Compiler
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since pkgimage /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so was not found
└ @ Base loading.jl:3945
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched
│   requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163]
│   cache file:      use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165]
└ @ Base loading.jl:3918
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since pkgimage /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so was not found
└ @ Base loading.jl:3945
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched
│   requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163]
│   cache file:      use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165]
└ @ Base loading.jl:3918
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since pkgimage /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so was not found
└ @ Base loading.jl:3945
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched
│   requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163]
│   cache file:      use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165]
└ @ Base loading.jl:3918
[ Info: Precompiling Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1] (cache misses: mismatched flags (2), missing ocachefile (2))
┌ Debug: Generating object cache file for Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1]
└ @ Base loading.jl:3021
┌ Debug: Loading object cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so for Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1]
└ @ Base loading.jl:1277
  1.000245 seconds (507.60 k allocations: 29.245 MiB, 1.68% gc time, 65.46% compilation time)

shell> ls -la /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so
-rwxrwxr-x 1 keno keno 7112 Nov  1 07:33 /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so

julia> Compiler === Base.Compiler
true
```

Next, we dev the Compiler from another Julia version. Now it just treats
is as a regular package load and generates a 5MB .ji that contains the usual
pkgimage.
```
(@v1.12) pkg> dev /home/keno/julia2/stdlib/Compiler
 Resolving package versions...
  Updating `~/.julia/environments/v1.12/Project.toml`
[807dbc54] + Compiler v0.0.1 `~/julia2/stdlib/Compiler`
  Updating `~/.julia/environments/v1.12/Manifest.toml`
[807dbc54] + Compiler v0.0.1 `~/julia2/stdlib/Compiler`

julia> using Compiler
┌ Debug: Rejecting stale cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji because file size of /home/keno/julia2/stdlib/Compiler/src/Compiler.jl has changed (file size 5958, before 5949)
└ @ Base loading.jl:3875
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(Base.UUID("807dbc54-b67e-4c79-8afb-eafe4df6f2e1"), "Compiler") since the flags are mismatched
│   requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163]
│   cache file:      use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165]
└ @ Base loading.jl:3918
┌ Debug: Rejecting stale cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.ji because file size of /home/keno/julia2/stdlib/Compiler/src/Compiler.jl has changed (file size 5958, before 5949)
└ @ Base loading.jl:3875
┌ Debug: Rejecting cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_ajVhJ.ji for Base.PkgId(nothing, "") since the flags are mismatched
│   requested flags: use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2 [163]
│   cache file:      use_pkgimages = true, debug_level = 2, check_bounds = 0, inline = true, opt_level = 2 [165]
└ @ Base loading.jl:3918
Precompiling Compiler finished.
  1 dependency successfully precompiled in 3 seconds
┌ Debug: Loading object cache file /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so for Compiler [807dbc54-b67e-4c79-8afb-eafe4df6f2e1]
└ @ Base loading.jl:1277

shell> ls -la /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so
-rwxrwxr-x 1 keno keno 5706792 Nov  1 07:40 /home/keno/.julia/compiled/v1.12/Compiler/DqelA_OpSbJ.so

julia> Compiler === Base.Compiler
false
```

Author: Keno

base/Base.jl

@@ -472,6 +472,18 @@ end
 # enable threads support
 @eval PCRE PCRE_COMPILE_LOCK = Threads.SpinLock()
 
+# Record dependency information for files beloning to the Compiler, so that
+# we know whether the .ji can just give the Base copy or not.
+# TODO: We may want to do this earlier to avoid TOCTOU issues.
+const _compiler_require_dependencies = Any[]
+for i = 1:length(_included_files)
+    isassigned(_included_files, i) || continue
+    (mod, file) = _included_files[i]
+    if parentmodule(mod) === Compiler || endswith(file, "/Compiler.jl")
+        _include_dependency!(_compiler_require_dependencies, true, mod, file, true, false)
+    end
+end
+
 end
 
 # Ensure this file is also tracked

base/loading.jl

@@ -2239,15 +2239,22 @@ const include_callbacks = Any[]
 const _concrete_dependencies = Pair{PkgId,UInt128}[] # these dependency versions are "set in stone", because they are explicitly loaded, and the process should try to avoid invalidating them
 const _require_dependencies = Any[] # a list of (mod, abspath, fsize, hash, mtime) tuples that are the file dependencies of the module currently being precompiled
 const _track_dependencies = Ref(false) # set this to true to track the list of file dependencies
-function _include_dependency(mod::Module, _path::AbstractString; track_content=true,
-                             path_may_be_dir=false)
+
+function _include_dependency(mod::Module, _path::AbstractString; track_content::Bool=true,
+                             path_may_be_dir::Bool=false)
+    _include_dependency!(_require_dependencies, _track_dependencies[], mod, _path, track_content, path_may_be_dir)
+end
+
+function _include_dependency!(dep_list::Vector{Any}, track_dependencies::Bool,
+                              mod::Module, _path::AbstractString,
+                              track_content::Bool, path_may_be_dir::Bool)
     prev = source_path(nothing)
     if prev === nothing
         path = abspath(_path)
     else
         path = normpath(joinpath(dirname(prev), _path))
     end
-    if !_track_dependencies[]
+    if !track_dependencies[]
         if !path_may_be_dir && !isfile(path)
             throw(SystemError("opening file $(repr(path))", Libc.ENOENT))
         elseif path_may_be_dir && !Filesystem.isreadable(path)
@@ -2258,9 +2265,9 @@ function _include_dependency(mod::Module, _path::AbstractString; track_content=t
             if track_content
                 hash = isdir(path) ? _crc32c(join(readdir(path))) : open(_crc32c, path, "r")
                 # use mtime=-1.0 here so that fsize==0 && mtime==0.0 corresponds to a missing include_dependency
-                push!(_require_dependencies, (mod, path, filesize(path), hash, -1.0))
+                push!(dep_list, (mod, path, filesize(path), hash, -1.0))
             else
-                push!(_require_dependencies, (mod, path, UInt64(0), UInt32(0), mtime(path)))
+                push!(dep_list, (mod, path, UInt64(0), UInt32(0), mtime(path)))
             end
         end
     end
@@ -2555,7 +2562,21 @@ end
 root_module_exists(key::PkgId) = @lock require_lock haskey(loaded_modules, key)
 maybe_root_module(key::PkgId) = @lock require_lock get(loaded_modules, key, nothing)
 
-loaded_modules_array() = @lock require_lock copy(loaded_modules_order)
+function loaded_modules_array()
+    @lock require_lock begin
+        ret = copy(loaded_modules_order)
+        if generating_output(false)
+            # If we're generating a non-incremental system image, also add the
+            # Compiler to the module list so that we generate native code for it.
+            # However, we do not add it to `loaded_modules_order` unless there
+            # is a Julia-level dependency on it. The (non-)loadedness of the compiler
+            # is an implementation detail and we do not want ordinary pkgimages to
+            # acquire a dependency on the Compiler if they do not use it internally.
+            push!(ret, Compiler)
+        end
+        return ret
+    end
+end
 
 # after unreference_module, a subsequent require call will try to load a new copy of it, if stale
 # reload(m) = (unreference_module(m); require(m))
@@ -3326,6 +3347,10 @@ mutable struct CacheHeaderIncludes
     const modpath::Vector{String}   # seemingly not needed in Base, but used by Revise
 end
 
+function CacheHeaderIncludes(dep_tuple::Tuple{Module, String, Int64, UInt32, Float64})
+    return CacheHeaderIncludes(PkgId(dep_tuple[1]), dep_tuple[2:end]..., String[])
+end
+
 function replace_depot_path(path::AbstractString, depots::Vector{String}=normalize_depots_for_relocation())
     for depot in depots
         if startswith(path, string(depot, Filesystem.pathsep())) || path == depot
@@ -3865,6 +3890,56 @@ function list_reasons(reasons::Dict{String,Int})
 end
 list_reasons(::Nothing) = ""
 
+function any_includes_stale(includes::Vector{CacheHeaderIncludes}, cachefile::String, reasons::Union{Dict{String,Int},Nothing}=nothing)
+    for chi in includes
+        f, fsize_req, hash_req, ftime_req = chi.filename, chi.fsize, chi.hash, chi.mtime
+        if startswith(f, string("@depot", Filesystem.pathsep()))
+            @debug("Rejecting stale cache file $cachefile because its depot could not be resolved")
+            record_reason(reasons, "nonresolveable depot")
+            return true
+        end
+        if !ispath(f)
+            _f = fixup_stdlib_path(f)
+            if _f != f && isfile(_f) && startswith(_f, Sys.STDLIB)
+                continue
+            end
+            @debug "Rejecting stale cache file $cachefile because file $f does not exist"
+            record_reason(reasons, "missing sourcefile")
+            return true
+        end
+        if ftime_req >= 0.0
+            # this is an include_dependency for which we only recorded the mtime
+            ftime = mtime(f)
+            is_stale = ( ftime != ftime_req ) &&
+                       ( ftime != floor(ftime_req) ) &&           # Issue #13606, PR #13613: compensate for Docker images rounding mtimes
+                       ( ftime != ceil(ftime_req) ) &&            # PR: #47433 Compensate for CirceCI's truncating of timestamps in its caching
+                       ( ftime != trunc(ftime_req, digits=6) ) && # Issue #20837, PR #20840: compensate for GlusterFS truncating mtimes to microseconds
+                       ( ftime != 1.0 )  &&                       # PR #43090: provide compatibility with Nix mtime.
+                       !( 0 < (ftime_req - ftime) < 1e-6 )        # PR #45552: Compensate for Windows tar giving mtimes that may be incorrect by up to one microsecond
+            if is_stale
+                @debug "Rejecting stale cache file $cachefile because mtime of include_dependency $f has changed (mtime $ftime, before $ftime_req)"
+                record_reason(reasons, "include_dependency mtime change")
+                return true
+            end
+        else
+            fstat = stat(f)
+            fsize = filesize(fstat)
+            if fsize != fsize_req
+                @debug "Rejecting stale cache file $cachefile because file size of $f has changed (file size $fsize, before $fsize_req)"
+                record_reason(reasons, "include_dependency fsize change")
+                return true
+            end
+            hash = isdir(fstat) ? _crc32c(join(readdir(f))) : open(_crc32c, f, "r")
+            if hash != hash_req
+                @debug "Rejecting stale cache file $cachefile because hash of $f has changed (hash $hash, before $hash_req)"
+                record_reason(reasons, "include_dependency fhash change")
+                return true
+            end
+        end
+    end
+    return false
+end
+
 # returns true if it "cachefile.ji" is stale relative to "modpath.jl" and build_id for modkey
 # otherwise returns the list of dependencies to also check
 @constprop :none function stale_cachefile(modpath::String, cachefile::String; ignore_loaded::Bool = false, requested_flags::CacheFlags=CacheFlags(), reasons=nothing)
@@ -4024,51 +4099,8 @@ end
                     return true
                 end
             end
-            for chi in includes
-                f, fsize_req, hash_req, ftime_req = chi.filename, chi.fsize, chi.hash, chi.mtime
-                if startswith(f, string("@depot", Filesystem.pathsep()))
-                    @debug("Rejecting stale cache file $cachefile because its depot could not be resolved")
-                    record_reason(reasons, "nonresolveable depot")
-                    return true
-                end
-                if !ispath(f)
-                    _f = fixup_stdlib_path(f)
-                    if _f != f && isfile(_f) && startswith(_f, Sys.STDLIB)
-                        continue
-                    end
-                    @debug "Rejecting stale cache file $cachefile because file $f does not exist"
-                    record_reason(reasons, "missing sourcefile")
-                    return true
-                end
-                if ftime_req >= 0.0
-                    # this is an include_dependency for which we only recorded the mtime
-                    ftime = mtime(f)
-                    is_stale = ( ftime != ftime_req ) &&
-                               ( ftime != floor(ftime_req) ) &&           # Issue #13606, PR #13613: compensate for Docker images rounding mtimes
-                               ( ftime != ceil(ftime_req) ) &&            # PR: #47433 Compensate for CirceCI's truncating of timestamps in its caching
-                               ( ftime != trunc(ftime_req, digits=6) ) && # Issue #20837, PR #20840: compensate for GlusterFS truncating mtimes to microseconds
-                               ( ftime != 1.0 )  &&                       # PR #43090: provide compatibility with Nix mtime.
-                               !( 0 < (ftime_req - ftime) < 1e-6 )        # PR #45552: Compensate for Windows tar giving mtimes that may be incorrect by up to one microsecond
-                    if is_stale
-                        @debug "Rejecting stale cache file $cachefile because mtime of include_dependency $f has changed (mtime $ftime, before $ftime_req)"
-                        record_reason(reasons, "include_dependency mtime change")
-                        return true
-                    end
-                else
-                    fstat = stat(f)
-                    fsize = filesize(fstat)
-                    if fsize != fsize_req
-                        @debug "Rejecting stale cache file $cachefile because file size of $f has changed (file size $fsize, before $fsize_req)"
-                        record_reason(reasons, "include_dependency fsize change")
-                        return true
-                    end
-                    hash = isdir(fstat) ? _crc32c(join(readdir(f))) : open(_crc32c, f, "r")
-                    if hash != hash_req
-                        @debug "Rejecting stale cache file $cachefile because hash of $f has changed (hash $hash, before $hash_req)"
-                        record_reason(reasons, "include_dependency fhash change")
-                        return true
-                    end
-                end
+            if any_includes_stale(includes, cachefile, reasons)
+                return true
             end
         end
 
@@ -4189,5 +4221,21 @@ end
 
 precompile(include_package_for_output, (PkgId, String, Vector{String}, Vector{String}, Vector{String}, typeof(_concrete_dependencies), Nothing)) || @assert false
 precompile(include_package_for_output, (PkgId, String, Vector{String}, Vector{String}, Vector{String}, typeof(_concrete_dependencies), String)) || @assert false
+<<<<<<< HEAD
 precompile(create_expr_cache, (PkgId, String, String, String, typeof(_concrete_dependencies), Cmd, CacheFlags, IO, IO)) || @assert false
 precompile(create_expr_cache, (PkgId, String, String, Nothing, typeof(_concrete_dependencies), Cmd, CacheFlags, IO, IO)) || @assert false
+=======
+precompile(create_expr_cache, (PkgId, String, String, String, typeof(_concrete_dependencies), Cmd, IO, IO)) || @assert false
+precompile(create_expr_cache, (PkgId, String, String, Nothing, typeof(_concrete_dependencies), Cmd, IO, IO)) || @assert false
+
+is_core_or_base(mod::Module) = mod === Core || mod === Base
+function prepare_compiler_stub_image!()
+    ccall(:jl_add_to_module_init_list, Cvoid, (Any,), Compiler)
+    # Drop any dependencies on loaded modules. We know Base.Compiler only depends
+    # on very basic `Base` functionality, so we don't want to pick up dependencies
+    # on other stdlibs that may be loaded.
+    filter!(p->is_core_or_base(p.second), loaded_modules)
+    register_root_module(Compiler)
+    filter!(is_core_or_base, loaded_modules_order)
+end
+>>>>>>> 84d9bc1466 (Short-cut parsing the compiler if you dev the loaded version)

contrib/generate_precompile.jl

@@ -359,6 +359,7 @@ generate_precompile_statements() = try # Make sure `ansi_enablecursor` is printe
             eval(PrecompileStagingArea, :(const $(Symbol(_mod)) = $_mod))
         end
     end
+    eval(PrecompileStagingArea, :(const Compiler = Base.Compiler))
 
     n_succeeded = 0
     # Make statements unique

src/module.c

@@ -1276,6 +1276,7 @@ JL_DLLEXPORT jl_uuid_t jl_module_uuid(jl_module_t* m) { return m->uuid; }
 
 // TODO: make this part of the module constructor and read-only?
 JL_DLLEXPORT void jl_set_module_uuid(jl_module_t *m, jl_uuid_t uuid) { m->uuid = uuid; }
+JL_DLLEXPORT void jl_set_module_parent(jl_module_t *m, jl_module_t *parent) { m->parent = parent; }
 
 int jl_is_submodule(jl_module_t *child, jl_module_t *parent) JL_NOTSAFEPOINT
 {
@@ -1308,15 +1309,20 @@ JL_DLLEXPORT void jl_clear_implicit_imports(jl_module_t *m)
     JL_UNLOCK(&m->lock);
 }
 
+JL_DLLEXPORT void jl_add_to_module_init_list(jl_value_t *mod)
+{
+    if (jl_module_init_order == NULL)
+        jl_module_init_order = jl_alloc_vec_any(0);
+    jl_array_ptr_1d_push(jl_module_init_order, mod);
+}
+
 JL_DLLEXPORT void jl_init_restored_module(jl_value_t *mod)
 {
     if (!jl_generating_output() || jl_options.incremental) {
         jl_module_run_initializer((jl_module_t*)mod);
     }
     else {
-        if (jl_module_init_order == NULL)
-            jl_module_init_order = jl_alloc_vec_any(0);
-        jl_array_ptr_1d_push(jl_module_init_order, mod);
+        jl_add_to_module_init_list(mod);
     }
 }
 

src/precompile_utils.c

@@ -281,6 +281,12 @@ static void *jl_precompile(int all)
     return native_code;
 }
 
+static int suppress_precompile = 0;
+JL_DLLEXPORT void jl_suppress_precompile(int suppress)
+{
+    suppress_precompile = suppress;
+}
+
 static void *jl_precompile_worklist(jl_array_t *worklist, jl_array_t *extext_methods, jl_array_t *new_ext_cis)
 {
     if (!worklist)
@@ -289,34 +295,36 @@ static void *jl_precompile_worklist(jl_array_t *worklist, jl_array_t *extext_met
     // type signatures that were inferred but haven't been compiled
     jl_array_t *m = jl_alloc_vec_any(0);
     JL_GC_PUSH1(&m);
-    size_t i, n = jl_array_nrows(worklist);
-    for (i = 0; i < n; i++) {
-        jl_module_t *mod = (jl_module_t*)jl_array_ptr_ref(worklist, i);
-        assert(jl_is_module(mod));
-        foreach_mtable_in_module(mod, precompile_enq_all_specializations_, m);
-    }
-    n = jl_array_nrows(extext_methods);
-    for (i = 0; i < n; i++) {
-        jl_method_t *method = (jl_method_t*)jl_array_ptr_ref(extext_methods, i);
-        assert(jl_is_method(method));
-        jl_value_t *specializations = jl_atomic_load_relaxed(&method->specializations);
-        if (!jl_is_svec(specializations)) {
-            precompile_enq_specialization_((jl_method_instance_t*)specializations, m);
+    if (!suppress_precompile) {
+        size_t i, n = jl_array_nrows(worklist);
+        for (i = 0; i < n; i++) {
+            jl_module_t *mod = (jl_module_t*)jl_array_ptr_ref(worklist, i);
+            assert(jl_is_module(mod));
+            foreach_mtable_in_module(mod, precompile_enq_all_specializations_, m);
         }
-        else {
-            size_t j, l = jl_svec_len(specializations);
-            for (j = 0; j < l; j++) {
-                jl_value_t *mi = jl_svecref(specializations, j);
-                if (mi != jl_nothing)
-                    precompile_enq_specialization_((jl_method_instance_t*)mi, m);
+        n = jl_array_nrows(extext_methods);
+        for (i = 0; i < n; i++) {
+            jl_method_t *method = (jl_method_t*)jl_array_ptr_ref(extext_methods, i);
+            assert(jl_is_method(method));
+            jl_value_t *specializations = jl_atomic_load_relaxed(&method->specializations);
+            if (!jl_is_svec(specializations)) {
+                precompile_enq_specialization_((jl_method_instance_t*)specializations, m);
+            }
+            else {
+                size_t j, l = jl_svec_len(specializations);
+                for (j = 0; j < l; j++) {
+                    jl_value_t *mi = jl_svecref(specializations, j);
+                    if (mi != jl_nothing)
+                        precompile_enq_specialization_((jl_method_instance_t*)mi, m);
+                }
             }
         }
-    }
-    if (new_ext_cis) {
-        n = jl_array_nrows(new_ext_cis);
-        for (i = 0; i < n; i++) {
-            jl_code_instance_t *ci = (jl_code_instance_t*)jl_array_ptr_ref(new_ext_cis, i);
-            precompile_enq_specialization_(ci->def, m);
+        if (new_ext_cis) {
+            n = jl_array_nrows(new_ext_cis);
+            for (i = 0; i < n; i++) {
+                jl_code_instance_t *ci = (jl_code_instance_t*)jl_array_ptr_ref(new_ext_cis, i);
+                precompile_enq_specialization_(ci->def, m);
+            }
         }
     }
     void *native_code = jl_precompile_(m, 1);

stdlib/Compiler/src/Compiler.jl

@@ -1,7 +1,31 @@
 # This file is a part of Julia. License is MIT: https://julialang.org/license
 
+# When generating an incremental precompile file, we first check whether we
+# already have a copy of this *exact* code in the system image. If so, we
+# simply generates a pkgimage that has the dependency edges we recorded in
+# the system image and simply returns that copy of the compiler. If not,
+# we proceed to load/precompile this as an ordinary package.
+if isdefined(Base, :generating_output) && Base.generating_output(true) &&
+        Base.samefile(Base._compiler_require_dependencies[1][2], @eval @__FILE__) &&
+        !Base.any_includes_stale(
+            map(Base.CacheHeaderIncludes, Base._compiler_require_dependencies),
+            "sysimg", nothing)
+
+    Base.prepare_compiler_stub_image!()
+    append!(Base._require_dependencies, Base._compiler_require_dependencies)
+    # There isn't much point in precompiling native code - downstream users will
+    # specialize their own versions of the compiler code and we don't activate
+    # the compiler by default anyway, so let's save ourselves some disk space.
+    ccall(:jl_suppress_precompile, Cvoid, (Cint,), 1)
 
-baremodule Compiler
+else
+
+@eval baremodule Compiler
+
+# Needs to match UUID defined in Project.toml
+ccall(:jl_set_module_uuid, Cvoid, (Any, NTuple{2, UInt64}), Compiler,
+    (0x807dbc54_b67e_4c79, 0x8afb_eafe4df6f2e1))
+ccall(:jl_set_module_parent, Cvoid, (Any, Any), Compiler, Compiler)
 
 using Core.Intrinsics, Core.IR
 
@@ -136,3 +160,5 @@ include("optimize.jl")
 include("bootstrap.jl")
 
 end
+
+end

stdlib/Makefile

@@ -43,7 +43,7 @@ $(foreach jll,$(JLLS),$(eval $(call download-artifacts-toml,$(jll))))
 STDLIBS = Artifacts Base64 CRC32c Dates FileWatching \
           Future InteractiveUtils Libdl LibGit2 LinearAlgebra Logging \
           Markdown Mmap Printf Profile Random REPL Serialization \
-          SharedArrays Sockets Test TOML Unicode UUIDs Compiler \
+          SharedArrays Sockets Test TOML Unicode UUIDs \
           $(JLL_NAMES)
 
 STDLIBS_EXT = Pkg Statistics LazyArtifacts LibCURL DelimitedFiles Downloads ArgTools \
@@ -56,7 +56,6 @@ ifneq ($(filter $(STDLIBS),$(STDLIBS_EXT)),)
 $(error ERROR duplicated STDLIBS in list)
 endif
 
-
 # Generate symlinks to all stdlibs at usr/share/julia/stdlib/vX.Y/
 $(foreach module, $(STDLIBS), $(eval $(call symlink_target,$$(JULIAHOME)/stdlib/$(module),$$(build_datarootdir)/julia/stdlib/$$(VERSDIR),$(module))))