Assertion/hang precompiling code that calls inference with custom AbstractInterpreter

[maleadt]

See #46296 (comment) for a MWE.

---

I was trying out SnoopPrecompile with CUDA.jl in an attempt to reduce TTFX, but doing so crashes Julia. I added the following to CUDA.jl's src/precompile.jl:

using SnoopPrecompile

@precompile_all_calls begin
    target = PTXCompilerTarget(; cap=v"7.0")
    params = CUDACompilerParams()
    job = CompilerJob(target, FunctionSpec(identity, Tuple{Nothing}, true), params)
    GPUCompiler.code_native(devnull, job)
end

On 1.8, this results in:

signal (11): Segmentation fault
in expression starting at none:1
jl_deserialize_value_singleton at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-8/src/dump.c:1941 [inlined]
jl_deserialize_value at /cache/build/default-amdci5-5/julialang/julia-release-1-dot-8/src/dump.c:2160

From a cursory look, it's inspecting dt->instance while dt is null. Reproducing this doesn't require a GPU, CUDA.jl can precompile without, but in any case here's an rr recording: https://s3.amazonaws.com/julialang-dumps/reports/2022-08-09T15-39-40-maleadt.tar.zst

I wanted to cut-out CUDA and reproduce this on GPUCompiler with a debug build, but I'm getting a different error adding the following to GPUCompiler's src/precompile.jl:

using SnoopPrecompile

struct CUDACompilerParams <: AbstractCompilerParams end

@precompile_all_calls begin
    target = PTXCompilerTarget(; cap=v"7.0")
    params = CUDACompilerParams()
    job = CompilerJob(target, FunctionSpec(identity, Tuple{Nothing}, true), params)
    GPUCompiler.code_native(devnull, job)
end

i.e. the same but with a dummy CUDACompilerParams definition. Trying to precompile GPUCompiler on 1.8 hangs, on 1.9 with a debug build it results in:

julia-debug: /home/tim/Julia/src/julia/src/dump.c:2159: jl_deserialize_value: Assertion `offs < backref_list.len' failed.

Maybe this is the same error? Again, here's an rr trace: https://s3.amazonaws.com/julialang-dumps/reports/2022-08-09T15-34-52-maleadt.tar.zst

[maleadt]

Reduced to the following:

/somedir/GPUCompiler.jl

module GPUCompiler

struct CodeCache
    dict
end
global_cache = CodeCache(Dict())
Core.Compiler.setindex!(cache, b, c) = push!(get(cache.dict, c, nothing), b)

struct Interpreter <: Core.Compiler.AbstractInterpreter
    cache
end
Core.Compiler.InferenceParams(i) = Core.Compiler.InferenceParams()
Core.Compiler.OptimizationParams(i) = Core.Compiler.OptimizationParams()
Core.Compiler.get_world_counter(i) = -1%UInt
Core.Compiler.get_inference_cache(i) = Vector()
Core.Compiler.code_cache(i) = Core.Compiler.WorldView(i.cache)
Core.Compiler.haskey(j, c) = Core.Compiler.get(j, c, nothing) !== nothing
function Core.Compiler.get(j, c, default)
    for b in get!(j.cache.dict, c, [])
    end
end
Core.Compiler.setindex!(j::Core.Compiler.WorldView, b, c) =
    Core.Compiler.setindex!(j.cache, b, c)

h = Tuple{typeof(identity), Tuple}
i = which(h)
j, env = ccall(:jl_type_intersection_with_env, Any, (Any, Any), h, h)
x = ccall(:jl_specializations_get_linfo, Ref{Core.MethodInstance},
          (Any, Any, Any, UInt), i, j, env, -1%UInt)
Core.Compiler.typeinf_ext_toplevel(Interpreter(global_cache), x)

end

On a debug build of latest master:

$ JULIA_LOAD_PATH=/somedir/ julia-debug -e 'using GPUCompiler.jl'
julia-debug: /home/tim/Julia/src/julia/src/dump.c:2159: jl_deserialize_value: Assertion `offs < backref_list.len' failed.

[61176] signal (6): Aborted
in expression starting at /home/tim/Julia/tools/creduce/main.jl:1
unknown function (ip: 0x7f3e3fd8f4dc)
gsignal at /usr/lib/libc.so.6 (unknown line)
abort at /usr/lib/libc.so.6 (unknown line)
unknown function (ip: 0x7f3e3fd2945b)
__assert_fail at /usr/lib/libc.so.6 (unknown line)

On 1.8 without assertions, this just hangs.

[maleadt]

Further reduced to:

module GPUCompiler
h = Tuple{typeof(identity), Tuple}
i = which(h)
j, env = ccall(:jl_type_intersection_with_env, Any, (Any, Any), h, h)
ccall(:jl_specializations_get_linfo, Ref{Core.MethodInstance},
      (Any, Any, Any, UInt), i, j, env, 1%UInt)
end

This crashes when adding some extra code to display the backrefs during deserialization. Looks like the table gets corrupted pretty early:

+ ./julia -t1 --startup-file=no -e 'using GPUCompiler'
DEBUG: Serializing GPUCompiler as backref 2
DEBUG: Serializing Symbol("#include") as backref 4
DEBUG: Serializing typeof(GPUCompiler.include) as backref 5
...
DEBUG: Deserializing backref 2/4... #<null>.GPUCompiler
DEBUG: Deserializing backref 4/5... Symbol("#include")
DEBUG: Deserializing backref 5/7...
[1789796] signal (11): Segmentation fault
in expression starting at none:1
...
ijl_static_show at /home/tim/Julia/src/julia/src/rtutils.c:1256
jl_deserialize_value at /home/tim/Julia/src/julia/src/dump.c:2166

I'll resume debugging next week, but if anybody (@vtjnash) has some thoughts, please let me know.

EDIT: this isn't valid; turns out not all deserialized objects are directly printable without deserialization completely finishing.