(Apple Silicon): crashes with Getting signal 16 received but handler not on signal stack

[vdayanand]

MWE:

tmp/yu julia --project=. 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pkg; Pkg.add("LibPQ");Pkg.add("LicenseCheck")
julia> using LibPQ
[ Info: Precompiling LibPQ [194296ae-ab2e-5f79-8cd4-7183a0a5a0d1]
julia> using LicenseCheck

julia> lic = find_license(<PATH to any source code folder>)

signal 16 received but handler not on signal stack
fatal error: non-Go code set up signal handler without SA_ONSTACK flag

runtime stack:
runtime.throw({0x13685f26b?, 0x1400004d680?})
	/opt/x86_64-linux-musl/go/src/runtime/panic.go:1047 +0x40 fp=0x1400029afe0 sp=0x1400029afb0 pc=0x1367d2580
runtime.sigNotOnStack(0x10)
	/opt/x86_64-linux-musl/go/src/runtime/signal_unix.go:1020 +0x5c fp=0x1400029b000 sp=0x1400029afe0 pc=0x1367e770c
runtime.adjustSignalStack(0x10, 0x1400004c000, 0x1400029b0a8)
	/opt/x86_64-linux-musl/go/src/runtime/signal_unix.go:581 +0x250 fp=0x1400029b070 sp=0x1400029b000 pc=0x1367e6a10
runtime.sigtrampgo(0x10, 0x1400029b210, 0x1400029b278)
	/opt/x86_64-linux-musl/go/src/runtime/signal_unix.go:469 +0x174 fp=0x1400029b0f0 sp=0x1400029b070 pc=0x1367e6664
runtime.sigtrampgo(0x10, 0x1400029b210, 0x1400029b278)
	<autogenerated>:1 +0x1c fp=0x1400029b120 sp=0x1400029b0f0 pc=0x1367ffd1c
runtime.sigtramp()
	/opt/x86_64-linux-musl/go/src/runtime/sys_darwin_arm64.s:227 +0x4c fp=0x1400029b1e0 sp=0x1400029b120 pc=0x1367ff2cc

goroutine 18 [syscall, locked to thread]:
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400005bfe0 sp=0x1400005bfe0 pc=0x1367fe534

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004a740 sp=0x1400004a720 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x1400004a7d0 sp=0x1400004a740 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004a7d0 sp=0x1400004a7d0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 17 [running, locked to thread]:
	goroutine running on other thread; stack unavailable

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000048fa0 sp=0x14000048f80 pc=0x1367d4ec4
runtime.goparkunlock(...)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.forcegchelper()
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:302 +0xb4 fp=0x14000048fd0 sp=0x14000048fa0 pc=0x1367d4d54
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x14000048fd0 sp=0x14000048fd0 pc=0x1367fe534
created by runtime.init.6
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:290 +0x24

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000049770 sp=0x14000049750 pc=0x1367d4ec4
runtime.goparkunlock(...)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/mgcsweep.go:297 +0x10c fp=0x140000497b0 sp=0x14000049770 pc=0x1367c2d7c
runtime.gcenable.func1()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:178 +0x28 fp=0x140000497d0 sp=0x140000497b0 pc=0x1367b73d8
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000497d0 sp=0x140000497d0 pc=0x1367fe534
created by runtime.gcenable
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:178 +0x70

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x5c591d?, 0x6553f100?, 0x0?, 0x0?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000049f50 sp=0x14000049f30 pc=0x1367d4ec4
runtime.goparkunlock(...)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x136c7a760)
	/opt/x86_64-linux-musl/go/src/runtime/mgcscavenge.go:389 +0x5c fp=0x14000049f80 sp=0x14000049f50 pc=0x1367c0cdc
runtime.bgscavenge(0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/mgcscavenge.go:622 +0xac fp=0x14000049fb0 sp=0x14000049f80 pc=0x1367c12dc
runtime.gcenable.func2()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:179 +0x28 fp=0x14000049fd0 sp=0x14000049fb0 pc=0x1367b7378
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x14000049fd0 sp=0x14000049fd0 pc=0x1367fe534
created by runtime.gcenable
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:179 +0xb4

goroutine 5 [finalizer wait]:
runtime.gopark(0x60000000000000?, 0x0?, 0x8?, 0x1?, 0x2000000000?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000048580 sp=0x14000048560 pc=0x1367d4ec4
runtime.goparkunlock(...)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:369
runtime.runfinq()
	/opt/x86_64-linux-musl/go/src/runtime/mfinal.go:180 +0x128 fp=0x140000487d0 sp=0x14000048580 pc=0x1367b65f8
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000487d0 sp=0x140000487d0 pc=0x1367fe534
created by runtime.createfing
	/opt/x86_64-linux-musl/go/src/runtime/mfinal.go:157 +0x94

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x136ca9bc0?, 0x3?, 0x85?, 0x45?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000044740 sp=0x14000044720 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140000447d0 sp=0x14000044740 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000447d0 sp=0x140000447d0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x136ca9bc0?, 0x1?, 0xc8?, 0x39?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004af40 sp=0x1400004af20 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x1400004afd0 sp=0x1400004af40 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004afd0 sp=0x1400004afd0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x12c5717ef949?, 0x3?, 0x3c?, 0x73?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004b740 sp=0x1400004b720 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x1400004b7d0 sp=0x1400004b740 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004b7d0 sp=0x1400004b7d0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x136ca9bc0?, 0x1?, 0xae?, 0xe4?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x1400004bf40 sp=0x1400004bf20 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x1400004bfd0 sp=0x1400004bf40 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x1400004bfd0 sp=0x1400004bfd0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x136ca9bc0?, 0x1?, 0x5f?, 0xed?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000044f40 sp=0x14000044f20 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x14000044fd0 sp=0x14000044f40 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x14000044fd0 sp=0x14000044fd0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x12c5717ef9f0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000045740 sp=0x14000045720 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140000457d0 sp=0x14000045740 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140000457d0 sp=0x140000457d0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x12c5717ef6af?, 0x3?, 0x1d?, 0x1e?, 0x0?)
	/opt/x86_64-linux-musl/go/src/runtime/proc.go:363 +0xe4 fp=0x14000508740 sp=0x14000508720 pc=0x1367d4ec4
runtime.gcBgMarkWorker()
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1235 +0xec fp=0x140005087d0 sp=0x14000508740 pc=0x1367b95cc
runtime.goexit()
	/opt/x86_64-linux-musl/go/src/runtime/asm_arm64.s:1165 +0x4 fp=0x140005087d0 sp=0x140005087d0 pc=0x1367fe534
created by runtime.gcBgMarkStartWorkers
	/opt/x86_64-linux-musl/go/src/runtime/mgc.go:1159 +0x28

[28939] signal (6): Abort trap: 6
in expression starting at REPL[3]:1
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 5961551 (Pool: 5954502; Big: 7049); GC: 8
[1]    28939 abort      julia --project=.

versioninfo (reproducible on latest version as well)

julia> versioninfo()
Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 8 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 6 virtual cores

[ericphanson]

I can reproduce on Julia v1.10.2; running the tests of this package in a loop tends to do it:

for i = 1:10000
              println(i); include("runtests.jl")
              end

Also if I press ctrl-t to send a siginfo, that will do it.

However, it does not seem to happen with nightly Julia. So I suspect it is a Julia bug that has been fixed, but I'm not sure when.

I thought at first it was JuliaLang/julia#51284, but I can trigger the issue on that commit, plus it's been backported anyway.

[staticfloat]

I just tried this with nightly Julia, and it occurs quite reliably there as well. @vtjnash is it possible for us to enable secondary stacks for all of our signal handlers?

[vtjnash]

Most (possibly all?) could easily run on the altstack without any issue

[staticfloat]

Well, I naively added SA_ONSTACK to every sa_flags I could find, and it didn't change the situation. I'm a little confused anyway, as it's complaining about signal 16, which is not one that we set a signal handler for, as far as I can tell.

[ericphanson]

Maybe some signal stuff needs to happen in https://github.com/JuliaPackaging/Yggdrasil/blob/master/L/licensecheck/bundled/main.go? (Ie could be my fault, not Julia’s)

[DilumAluthge]

I am hitting this relatively frequently (but non-deterministically) when running the tests of PackageAnalyzer.jl on Apple Silicon.

[DilumAluthge]

Did we ever figure out what was going on here? I'm still hitting this relatively frequently on Apple Silicon. I'm seeing it on both Julia 1.10 and Julia nightly (1.13) - so if it was previously working on nightly, it no longer seems to be the case.

[ericphanson]

No, it’s a mystery to me