Stuck on precompiling after upgrading to 1.8

[polhager]

Hello! After upgrading to Julia 1.8, one of my machines is refusing to precompile TriangularSolve. ] add TriangularSolve (or any other package that has this a dependency) results in the REPL getting stuck at

Precompiling project...
  Progress [>                                        ]  0/1
  ◑ TriangularSolve

I've let it sit for an hour, and nothing changes. On 1.7, I do not experience this issue. I've tried doing a fresh install of julia, to no avail. Interrupting with ctrl+c gives the following

ERROR: The following 1 direct dependency failed to precompile:

TriangularSolve [d5829a12-d9aa-46ab-831f-fb7c9ab06edf]

Failed to precompile TriangularSolve [d5829a12-d9aa-46ab-831f-fb7c9ab06edf] to /home/philip/.julia/compiled/v1.8/TriangularSolve/jl_uahkjT.

signal (2): Interrupt
in expression starting at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:535
rdiv_block_N! at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:380
rdiv_block_MandN! at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:416
div_dispatch! at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:312
rdiv! at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:323
unknown function (ip: 0x7f3459fe5b71)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
macro expansion at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:543 [inlined]
macro expansion at /home/philip/.julia/packages/SnoopPrecompile/UWvXF/src/SnoopPrecompile.jl:51 [inlined]
macro expansion at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:540 [inlined]
macro expansion at /home/philip/.julia/packages/SnoopPrecompile/UWvXF/src/SnoopPrecompile.jl:107 [inlined]
top-level scope at /home/philip/.julia/packages/TriangularSolve/E948Y/src/TriangularSolve.jl:536
jl_toplevel_eval_flex at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:897
jl_eval_module_expr at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:203 [inlined]
jl_toplevel_eval_flex at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:709
jl_toplevel_eval_flex at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:850
ijl_toplevel_eval_in at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1428
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
_include at ./loading.jl:1488
include at ./Base.jl:419 [inlined]
include_package_for_output at ./loading.jl:1554
jfptr_include_package_for_output_37424.clone_1 at /home/philip/packages/julias/julia-1.8/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
do_call at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/interpreter.c:126
eval_value at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/interpreter.c:215
eval_stmt_value at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/interpreter.c:166 [inlined]
eval_body at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/interpreter.c:612
jl_interpret_toplevel_thunk at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/interpreter.c:750
jl_toplevel_eval_flex at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:906
jl_toplevel_eval_flex at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:850
ijl_toplevel_eval_in at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1428
include_string at ./loading.jl:1438
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
exec_options at ./client.jl:301
_start at ./client.jl:522
jfptr__start_30463.clone_1 at /home/philip/packages/julias/julia-1.8/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
true_main at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/cli/loader_exe.c:59
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
unknown function (ip: (nil))
Allocations: 26634325 (Pool: 26620513; Big: 13812); GC: 27

Curiously, only one of my machines experiences this issue. The other could upgrade to 1.8 with no issues. Both are Ubuntu 20.04 VMs running on win10 hosts. versioninfo() gives

julia> versioninfo()
Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 6 × AMD Ryzen 5 3600 6-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, znver2)
  Threads: 6 on 6 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS = 6

In addition, I have the same issue with v0.1.12, although with a slightly different stacktrace upon ctrl+c.

[chriselrod]

Precompilation was mostly useless for TriangularSolve before Julia 1.8, so it only precompiles with Julia versions newer than 1.8.0-beta1.

TriangularSolve.jl/src/TriangularSolve.jl

Lines 535 to 553 in 639930a

	@static if VERSION >= v"1.8.0-beta1"
	@precompile_setup begin
	A = rand(1, 1)
	B = rand(1, 1)
	res = similar(A)
	@precompile_all_calls begin
	rdiv!(res, A, UpperTriangular(B))
	rdiv!(res, A, UnitUpperTriangular(B))
	rdiv!(res, A, UpperTriangular(B), Val(false))
	rdiv!(res, A, UnitUpperTriangular(B), Val(false))
	__init__()
	ldiv!(res, LowerTriangular(B), A)
	ldiv!(res, UnitLowerTriangular(B), A)
	ldiv!(res, LowerTriangular(B), A, Val(false))
	ldiv!(res, UnitLowerTriangular(B), A, Val(false))
	end
	end
	end

That's obviously where it's getting stuck.
What happens if you dev TriangularSolve, comment out these lines, and then try running

using TriangularSolve
A = rand(1, 1) 
B = rand(1, 1) 
res = similar(A) 
TriangularSolve.rdiv!(res, A, UpperTriangular(B)) 
TriangularSolve.rdiv!(res, A, UnitUpperTriangular(B)) 
TriangularSolve.rdiv!(res, A, UpperTriangular(B), Val(false)) 
TriangularSolve.rdiv!(res, A, UnitUpperTriangular(B), Val(false)) 
TriangularSolve.ldiv!(res, LowerTriangular(B), A) 
TriangularSolve.ldiv!(res, UnitLowerTriangular(B), A) 
TriangularSolve.ldiv!(res, LowerTriangular(B), A, Val(false)) 
TriangularSolve.ldiv!(res, UnitLowerTriangular(B), A, Val(false)) 

I'm assuming these work from the REPL?

It seems it was getting stuck in that first rdiv! call.

[polhager]

julia> rdiv!(res, A, UpperTriangular(B))
ERROR: MethodError: no method matching rdiv!(::Matrix{Float64}, ::Matrix{Float64}, ::UpperTriangular{Float64, Matrix{Float64}})
Closest candidates are:
  rdiv!(::StridedMatrix{T}, ::LowerTriangular{T, <:StridedMatrix{T} where T}) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/packages/julias/julia-1.8/share/julia/stdlib/v1.8/LinearAlgebra/src/triangular.jl:733
  rdiv!(::StridedMatrix{T}, ::UnitLowerTriangular{T, <:StridedMatrix{T} where T}) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/packages/julias/julia-1.8/share/julia/stdlib/v1.8/LinearAlgebra/src/triangular.jl:733
  rdiv!(::StridedMatrix{T}, ::UpperTriangular{T, <:StridedMatrix{T} where T}) where T<:Union{Float32, Float64, ComplexF32, ComplexF64} at ~/packages/julias/julia-1.8/share/julia/stdlib/v1.8/LinearAlgebra/src/triangular.jl:733
  ...
Stacktrace:
 [1] top-level scope
   @ REPL[26]:1

Doing TriangularSolve.rdiv!(res, A, UpperTriangular(B)) causes it to think for a couple of seconds and then

julia> TriangularSolve.rdiv!(res, A, UpperTriangular(B))

signal (11): Segmentation fault
in expression starting at none:0
ijl_types_equal at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/subtype.c:1927
jl_specializations_get_linfo_ at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:129
cache_method at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:1091
ml_matches at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:3266
_gf_invoke_lookup at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2561 [inlined]
jl_mt_assoc_by_type at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:1249
jl_lookup_generic_ at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2520 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2545
#current_exceptions#79 at ./error.jl:157
current_exceptions at ./error.jl:150 [inlined]
current_exceptions at ./error.jl:150 [inlined]
eval_user_input at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:163
repl_backend_loop at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
start_repl_backend at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
#run_repl#47 at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
run_repl at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
jfptr_run_repl_67214.clone_1 at /home/philip/packages/julias/julia-1.8/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
#966 at ./client.jl:419
jfptr_YY.966_46273.clone_1 at /home/philip/packages/julias/julia-1.8/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
jl_f__call_latest at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/builtins.c:774
#invokelatest#2 at ./essentials.jl:729 [inlined]
invokelatest at ./essentials.jl:726 [inlined]
run_main_repl at ./client.jl:404
exec_options at ./client.jl:318
_start at ./client.jl:522
jfptr__start_30463.clone_1 at /home/philip/packages/julias/julia-1.8/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2367 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/gf.c:2549
jl_apply at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/julia.h:1838 [inlined]
true_main at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/jlapi.c:575
jl_repl_entrypoint at /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/src/jlapi.c:719
main at julia (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 21826055 (Pool: 21814262; Big: 11793); GC: 20
Segmentation fault (core dumped)

[chriselrod]

Ah, yes, rdiv! and ldiv! aren't exported so they don't conflict with LinearAlgebra, and also require explicit TriangularSolve. to document where the call came from.

Maybe you could get an rr trace of this segfault?
It looks like it is coming from the compiler, so maybe someone from the compiler team can take a look at what's going on.

[polhager]

What's an rr trace? :D And who do I send it to?

[chriselrod]

https://julialang.org/blog/2020/05/rr/
I guess you could share the link here, but I don't actually have any experience using rr, nor the time to learn now or fix this.

[polhager]

Doing julia --bug-report=rr results in

[ Info: Loading BugReporting package...
rr: Saving execution to trace directory `/home/philip/.julia/artifacts/jl_bIrxA3/julia-0'.
[FATAL src/PerfCounters.cc:306:start_counter()] Unable to open performance counter with 'perf_event_open'; are hardware perf events available? See https://github.com/rr-debugger/rr/wiki/Will-rr-work-on-my-system
┌ Error: Debugged process failed
│   exitcode = 1
│   termsignal = 0
└ @ BugReporting ~/.julia/packages/BugReporting/AZD4C/src/BugReporting.jl:191

I think we're out of luck on this: at https://github.com/rr-debugger/rr/wiki/Building-And-Installing

VirtualBox does not work at this time because it doesn't support PMU virtualization.

...and I'm using VirtualBox. So no rr trace.

[markowkes]

@polhager Did you find a solution to this? I'm seeing the same issue with Julia 1.8.1 on Windows10 running through parallels on a Mac with ARM.

[polhager]

@markowkes Not really. My bandaid solution is to just use a local copy (] dev TriangularSolve) and comment out the precompilation lines

using SnoopPrecompile
@static if VERSION >= v"1.8.0-beta1"
  @precompile_setup begin
    A = rand(1, 1)
    B = rand(1, 1)
    res = similar(A)
    @precompile_all_calls begin
      rdiv!(res, A, UpperTriangular(B))
      rdiv!(res, A, UnitUpperTriangular(B))
      rdiv!(res, A, UpperTriangular(B), Val(false))
      rdiv!(res, A, UnitUpperTriangular(B), Val(false))

      __init__()
      ldiv!(res, LowerTriangular(B), A)
      ldiv!(res, UnitLowerTriangular(B), A)
      ldiv!(res, LowerTriangular(B), A, Val(false))
      ldiv!(res, UnitLowerTriangular(B), A, Val(false))
    end
  end
end

[chriselrod]

Could you try if it works if you only comment out the versions without Val(false)?
E.g., what if you use

using SnoopPrecompile
@static if VERSION >= v"1.8.0-beta1"
  @precompile_setup begin
    A = rand(1, 1)
    B = rand(1, 1)
    res = similar(A)
    @precompile_all_calls begin
      rdiv!(res, A, UpperTriangular(B), Val(false))
      rdiv!(res, A, UnitUpperTriangular(B), Val(false))

      __init__()
      ldiv!(res, LowerTriangular(B), A, Val(false))
      ldiv!(res, UnitLowerTriangular(B), A, Val(false))
    end
  end
end

?
If this works, I'll switch it to this, and maybe make Val(false) the default.

[polhager]

That did not work, however commenting out the ones WITH Val(false) did work. So currently

@static if VERSION >= v"1.8.0-beta1"
 @precompile_setup begin
   A = rand(1, 1)
   B = rand(1, 1)
   res = similar(A)
   @precompile_all_calls begin
     rdiv!(res, A, UpperTriangular(B))
     rdiv!(res, A, UnitUpperTriangular(B))

     __init__()
     ldiv!(res, LowerTriangular(B), A)
     ldiv!(res, UnitLowerTriangular(B), A)
   end
 end
end

is working.

[chriselrod]

Oops, didn't mean to close this, as I left the precompile statements in.