Failed assertion during IR decompression

[maleadt]

Introduced very recently, likely in #56880

julia: /source/src/julia.h:1319: jl_array_ptr_ref: Assertion `i < ((*(size_t*)jl_svecref(((jl_datatype_t*)(((((jl_taggedvalue_t*)((char*)(a) - sizeof(jl_taggedvalue_t)))->header) & ~(uintptr_t)15)))->parameters, 1)) == 1 ? (((jl_array_t*)(a))->dimsize[0]) : (((jl_array_t*)(a))->ref.mem->length))' failed.

[278] signal 6 (-6): Aborted
in expression starting at /home/pkgeval/.julia/packages/PyCall/1gn3u/test/runtests.jl:36
unknown function (ip: 0x7b80c81afd3c) at /lib/x86_64-linux-gnu/libc.so.6
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7b80c814b394) at /lib/x86_64-linux-gnu/libc.so.6
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
jl_array_ptr_ref at /source/src/julia.h:1319
jl_array_ptr_ref at /source/src/julia.h:1319 [inlined]
lookup_root at /source/src/method.c:1483
jl_decode_value at /source/src/ircode.c:807
jl_decode_value_expr at /source/src/ircode.c:682 [inlined]
jl_decode_value at /source/src/ircode.c:840
ijl_uncompress_ir at /source/src/ircode.c:1158
_uncompressed_ir at ./runtime_internals.jl:1402
typeinf_ext_toplevel at ./../usr/share/julia/Compiler/src/typeinfer.jl:1215
jfptr_typeinf_ext_toplevel_117523.1 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /source/src/gf.c:3444 [inlined]
ijl_apply_generic at /source/src/gf.c:3644
jl_apply at /source/src/julia.h:2246 [inlined]
jl_type_infer at /source/src/gf.c:452
jl_compile_method_internal at /source/src/gf.c:2942
_jl_invoke at /source/src/gf.c:3436 [inlined]
ijl_apply_generic at /source/src/gf.c:3644
jl_apply at /source/src/julia.h:2246 [inlined]
do_call at /source/src/interpreter.c:125
eval_value at /source/src/interpreter.c:243
eval_stmt_value at /source/src/interpreter.c:194 [inlined]
eval_body at /source/src/interpreter.c:687
eval_body at /source/src/interpreter.c:562
eval_body at /source/src/interpreter.c:562

As seen on PkgEval: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2025-01/05/report.html

rr traces are available:

• https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/rr/PyCall-1736070706.tar.zst
• https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/rr/JuliaInterpreter-1736062570.tar.zst

[vtjnash]

I can't reproduce these, so I assume #56975 fixes it

[maleadt]

Latest PkgEval contains that commit, and there's still assertion failures during IR decompression, albeit slightly differently.

e.g. FileIO: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2025-01/09/FileIO.primary.log

julia: /source/src/method.c:1474: lookup_root: Assertion `key == 0' failed.

[21] signal 6 (-6): Aborted
in expression starting at /home/pkgeval/.julia/packages/FileIO/PtqMQ/test/query.jl:51
unknown function (ip: 0x7af52ab40ebc) at /lib/x86_64-linux-gnu/libc.so.6
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7af52aadc394) at /lib/x86_64-linux-gnu/libc.so.6
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
lookup_root at /source/src/method.c:1474
jl_decode_value at /source/src/ircode.c:805
jl_decode_value_memory at /source/src/ircode.c:599
jl_decode_value_array1d at /source/src/ircode.c:642 [inlined]
jl_decode_value at /source/src/ircode.c:831
ijl_uncompress_ir at /source/src/ircode.c:1161
_uncompressed_ir at ./runtime_internals.jl:1404

I cannot reproduce this specific one though, so maybe one of the subsequent commits on master fixed this already.

[vchuravy]

That one is weird. It looks similar to #56947 which was fixed by #54738

[maleadt]

Still happens on 090e291: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2025-01/22/report.html

[maleadt]

The error has slightly changed, but more packages are affected:

julia: /source/src/method.c:1411: lookup_root: Assertion `key == 0' failed.

[21] signal 6 (-6): Aborted
in expression starting at /home/pkgeval/.julia/packages/FileIO/PtqMQ/test/query.jl:51
unknown function (ip: 0x79154923febc) at /lib/x86_64-linux-gnu/libc.so.6
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7915491db394) at /lib/x86_64-linux-gnu/libc.so.6
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
lookup_root at /source/src/method.c:1411
jl_decode_value at /source/src/ircode.c:805
jl_decode_value_memory at /source/src/ircode.c:599
jl_decode_value_array1d at /source/src/ircode.c:642 [inlined]
jl_decode_value at /source/src/ircode.c:831
ijl_uncompress_ir at /source/src/ircode.c:1161
_uncompressed_ir at ./runtime_internals.jl:1413
typeinf_ext_toplevel at ./../usr/share/julia/Compiler/src/typeinfer.jl:1241
jfptr_typeinf_ext_toplevel_118592.1 at /opt/julia/lib/julia/sys.so (unknown line)
_jl_invoke at /source/src/gf.c:3474 [inlined]
ijl_apply_generic at /source/src/gf.c:3674

I still cannot reproduce, but all of the crashes have rr traces, e.g., https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/rr/FileIO-1737876863.tar.zst for FileIO.jl. These are replayable on any of the amdci servers.

[serenity4]

I haven't yet managed to reproduce it, but a quick look at the rr trace linked in the previous comment revealed that (at least for one crash instance, on FileIO tests) the method in question is

julia/base/boot.jl

Line 1046 in 162cfb3

Pair(a, b) = new{typeof(a), typeof(b)}(a, b)

which doesn't seem to actually say much. The stack trace further indicates that the error occurred while decoding ssavaluetypes. The key triggering this assertion has value 4579015910496103.

Just in case, does PkgEval run tests in a specific way, or does it simply use Pkg.test()?

[maleadt]

Just in case, does PkgEval run tests in a specific way, or does it simply use Pkg.test()?

The driver is somewhat complicated: https://github.com/JuliaCI/PkgEval.jl/blob/master/scripts/evaluate.jl

Some of the nonstandard things is that we run with --pkgimages=existing, while first using TestEnv.jl to precompile all test deps. Also, the build defaults to enabling assertions.

[serenity4]

It seems to be related to pkgimages.

I was able to reproduce locally on FileIO with --pkgimages=no and with --pkgimages=existing on an assert build. Using no makes it easier not to accidentally use code that is already cached, which may otherwise prevent the error from occurring.

I was able to reproduce both with Pkg.test("FileIO"; julia_args=["--pkgimages=no"]) and within a local session instantiating test dependencies via TestEnv.

[maleadt]

Reproducer: run the following with --pkgimages=no

using FileIO
module LoadTest1
    import FileIO: @format_str, File
    load(file::File{format"MultiLib"}) = error()
    save(file::File{format"MultiLib"}, data) = open(file, "w") do s
        write(s, magic(format"MultiLib"))  # Write the magic bytes
        write(s, 0)
    end
end
add_format(format"FUNCTION_FOR_MAGIC_BYTES", io -> true, ".wav", [LoadTest1])
del_format(format"FUNCTION_FOR_MAGIC_BYTES")

Example shell session:

❯ ./julia --pkgimages=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.12.0-DEV.1945 (2025-01-28)
 _/ |\__'_|_|_|\__'_|  |  Commit 575d8e805d* (8 days old master)
|__/                   |

julia> using Pkg

julia> Pkg.activate(;temp=true)

julia> Pkg.add("FileIO")
  [5789e2e9] + FileIO v1.16.6
  [ae029012] + Requires v1.3.0
  [0dad84c5] + ArgTools v1.1.2
  [56f22d72] + Artifacts v1.11.0
  [2a0f44e3] + Base64 v1.11.0
  [ade2ca70] + Dates v1.11.0
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching v1.11.0
  [dc6e5ff7] + JuliaSyntaxHighlighting v1.12.0
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2 v1.11.0
  [8f399da3] + Libdl v1.11.0
  [56ddb016] + Logging v1.11.0
  [d6f4376e] + Markdown v1.11.0
  [ca575930] + NetworkOptions v1.3.0
  [44cfe95a] + Pkg v1.12.0
  [de0858da] + Printf v1.11.0
  [9a3f8284] + Random v1.11.0
  [ea8e919c] + SHA v0.7.0
  [f489334b] + StyledStrings v1.11.0
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs v1.11.0
  [4ec0a83e] + Unicode v1.11.0
  [deac9b47] + LibCURL_jll v8.11.1+1
  [e37daf67] + LibGit2_jll v1.9.0+0
  [29816b5a] + LibSSH2_jll v1.11.3+1
  [14a3606d] + MozillaCACerts_jll v2024.12.31
  [458c3c95] + OpenSSL_jll v3.0.15+2
  [83775a58] + Zlib_jll v1.3.1+2
  [8e850ede] + nghttp2_jll v1.64.0+1
  [3f19e933] + p7zip_jll v17.5.0+2

julia> using FileIO

julia> module LoadTest1
           import FileIO: @format_str, File
           load(file::File{format"MultiLib"}) = error()
           save(file::File{format"MultiLib"}, data) = open(file, "w") do s
               write(s, magic(format"MultiLib"))  # Write the magic bytes
               write(s, 0)
           end
       end

julia> add_format(format"FUNCTION_FOR_MAGIC_BYTES", io -> true, ".wav", [LoadTest1])
julia: /home/tim/Julia/src/julia/src/method.c:1411: lookup_root: Assertion `key == 0' failed.

I'm seeing if I can further reduce this to an MWE.