runtime: failures with `fatal error: traceback did not unwind completely` on `openbsd/arm64`

[bcmills]

#!watchflakes
post <- goos == "openbsd" && goarch == "arm64" && log ~ `fatal error: traceback did not unwind completely`

(Pulled out from #55167 (comment), #62105 (comment), and #62101 (comment).)

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: traceback did not unwind completely`

2023-08-07 19:31 linux-arm64-longtest go@d367ec6a runtime.TestVDSO (log)

--- FAIL: TestVDSO (0.65s)
    crash_test.go:158: /tmp/workdir/tmp/go-build1177904178/testprog.exe SignalInVDSO: exit status 2
    crash_test.go:161: output:
        runtime: g1: frame.sp=0x400029af2f top=0x400029afd0
        	stack=[0x400029a000-0x400029b000
        fatal error: traceback did not unwind completely

        runtime stack:
        runtime.throw({0x10ccf6?, 0x1?})
        	/tmp/workdir/go/src/runtime/panic.go:859 +0x40 fp=0xfffff3b14370 sp=0xfffff3b14340 pc=0x4a230
        runtime.(*unwinder).finishInternal(0xfffff3b14488?)
        	/tmp/workdir/go/src/runtime/traceback.go:561 +0x138 fp=0xfffff3b143b0 sp=0xfffff3b14370 pc=0x710e8
        runtime.(*unwinder).next(0xfffff3b14488?)
        	/tmp/workdir/go/src/runtime/traceback.go:442 +0x2cc fp=0xfffff3b14440 sp=0xfffff3b143b0 pc=0x70efc
        runtime.copystack(0x40000021a0, 0x1000)
        	/tmp/workdir/go/src/runtime/stack.go:931 +0x30c fp=0xfffff3b14840 sp=0xfffff3b14440 pc=0x65c2c
        runtime.newstack()
        	/tmp/workdir/go/src/runtime/stack.go:1113 +0x384 fp=0xfffff3b149f0 sp=0xfffff3b14840 pc=0x660f4
        runtime.morestack()
        	/tmp/workdir/go/src/runtime/asm_arm64.s:316 +0x70 fp=0xfffff3b149f0 sp=0xfffff3b149f0 pc=0x7d5d0


        wanted:
        success

2023-08-07 19:31 linux-arm64-longtest go@aca65771 runtime.TestCgoCCodeSIGPROF (log)

--- FAIL: TestCgoCCodeSIGPROF (0.28s)
    crash_cgo_test.go:217: /tmp/workdir/tmp/go-build2962622675/testprogcgo.exe CgoCCodeSIGPROF: exit status 2
    crash_cgo_test.go:220: expected "OK\n" got runtime: g6: frame.sp=0x4000284782 top=0x40002847d0
        	stack=[0x4000284000-0x4000284800
        fatal error: traceback did not unwind completely

        runtime stack:
        runtime.throw({0x572985?, 0x1?})
        	/tmp/workdir/go/src/runtime/panic.go:859 +0x40 fp=0xf8d5df7fb400 sp=0xf8d5df7fb3d0 pc=0x43dc00
        runtime.(*unwinder).finishInternal(0xf8d5df7fb518?)
        	/tmp/workdir/go/src/runtime/traceback.go:558 +0x138 fp=0xf8d5df7fb440 sp=0xf8d5df7fb400 pc=0x468c98
        runtime.(*unwinder).next(0xf8d5df7fb518?)
        	/tmp/workdir/go/src/runtime/traceback.go:439 +0x2c8 fp=0xf8d5df7fb4d0 sp=0xf8d5df7fb440 pc=0x468aa8
        runtime.copystack(0x4000003d40, 0x800)
        	/tmp/workdir/go/src/runtime/stack.go:930 +0x2fc fp=0xf8d5df7fb5d0 sp=0xf8d5df7fb4d0 pc=0x45b5bc
        runtime.newstack()
        	/tmp/workdir/go/src/runtime/stack.go:1112 +0x384 fp=0xf8d5df7fb780 sp=0xf8d5df7fb5d0 pc=0x45ba84
        runtime.morestack()
        	/tmp/workdir/go/src/runtime/asm_arm64.s:316 +0x70 fp=0xf8d5df7fb780 sp=0xf8d5df7fb780 pc=0x476ab0

2023-08-15 03:40 plan9-arm go@60506f4d (log)

runtime: signal received on thread not created by Go.
runtime: g820: frame.sp=0x9a37e24 top=0x9a37fec
	stack=[0x9a28000-0x9a38000
fatal error: traceback did not unwind completely

runtime stack:
go tool dist: Failed: exit status: 'go 43241: runtime: signal received on thread not created by Go.
'
runtime.throw({0x68c649, 0x23})
	runtime/panic.go:859 +0x4c fp=0x3ffff78c sp=0x3ffff778 pc=0x40fb0
runtime.(*unwinder).finishInternal(0x3ffff824)
	runtime/traceback.go:561 +0x164 fp=0x3ffff7ac sp=0x3ffff78c pc=0x691bc
runtime.(*unwinder).next(0x3ffff824)
	runtime/traceback.go:442 +0x270 fp=0x3ffff804 sp=0x3ffff7ac pc=0x68e40
runtime.copystack(0x9576500, 0x10000)
	runtime/stack.go:931 +0x2c0 fp=0x3ffffa78 sp=0x3ffff804 pc=0x5aab8
runtime.shrinkstack(0x9576500)
	runtime/stack.go:1215 +0x118 fp=0x3ffffa88 sp=0x3ffffa78 pc=0x5bb0c
runtime.scanstack(0x9576500, 0x943295c)
	runtime/mgcmark.go:776 +0x128 fp=0x3ffffd30 sp=0x3ffffa88 pc=0x26144
runtime.markroot.func1()
	runtime/mgcmark.go:240 +0xf4 fp=0x3ffffd5c sp=0x3ffffd30 pc=0x24a44
runtime.markroot(0x943295c, 0x3d, 0x1)
	runtime/mgcmark.go:214 +0x234 fp=0x3ffffdb8 sp=0x3ffffd5c pc=0x24620
runtime.gcDrain(0x943295c, 0x3)
	runtime/mgcmark.go:1069 +0x5ac fp=0x3ffffe04 sp=0x3ffffdb8 pc=0x27088
runtime.gcBgMarkWorker.func2()
	runtime/mgc.go:1386 +0xb8 fp=0x3ffffe30 sp=0x3ffffe04 pc=0x21cd0
runtime.systemstack()
	runtime/asm_arm.s:317 +0x60 fp=0x3ffffe34 sp=0x3ffffe30 pc=0x754e0

2023-08-20 05:56 openbsd-arm64-jsing go@a2ec545e os/signal.test [build] (log)

go.go(/home/gopher/build/tmp/go-link-917651527/go.o:(syscall.libc_syscall_trampoline.abi0)): warning: syscall() may go away, please rewrite code to use direct calls

2023-08-20 05:56 openbsd-arm64-jsing go@a2ec545e strings (log)

runtime: g1: frame.sp=0x400011fe00 top=0x400013ffd0
	stack=[0x4000100000-0x4000140000
fatal error: traceback did not unwind completely

runtime stack:
runtime.throw({0x171b13?, 0x400013ffb0?})
	/home/gopher/build/go/src/runtime/panic.go:1018 +0x40 fp=0x4dff20360 sp=0x4dff20330 pc=0x452a0
runtime.(*unwinder).finishInternal(0x202f0?)
	/home/gopher/build/go/src/runtime/traceback.go:561 +0x110 fp=0x4dff203a0 sp=0x4dff20360 pc=0x6b7a0
runtime.(*unwinder).next(0x4dff20478?)
	/home/gopher/build/go/src/runtime/traceback.go:442 +0x2a4 fp=0x4dff20430 sp=0x4dff203a0 pc=0x6b5e4
runtime.scanstack(0x40000021a0, 0x7?)
	/home/gopher/build/go/src/runtime/mgcmark.go:802 +0x23c fp=0x4dff20870 sp=0x4dff20430 pc=0x2d8dc
runtime.markroot.func1()
	/home/gopher/build/go/src/runtime/mgcmark.go:240 +0xb4 fp=0x4dff208c0 sp=0x4dff20870 pc=0x2c664
runtime.markroot(0x400002a140, 0x14, 0x1)
	/home/gopher/build/go/src/runtime/mgcmark.go:214 +0x1cc fp=0x4dff20970 sp=0x4dff208c0 pc=0x2c32c
runtime.gcDrain(0x400002a140, 0x7)
	/home/gopher/build/go/src/runtime/mgcmark.go:1069 +0x3f8 fp=0x4dff209d0 sp=0x4dff20970 pc=0x2e308
runtime.gcBgMarkWorker.func2()
	/home/gopher/build/go/src/runtime/mgc.go:1405 +0x74 fp=0x4dff20a20 sp=0x4dff209d0 pc=0x2a504
runtime.systemstack()
	/home/gopher/build/go/src/runtime/asm_arm64.s:243 +0x6c fp=0x4dff20a30 sp=0x4dff20a20 pc=0x765bc

— watchflakes

[bcmills]

Hrm. The failure on linux-arm64-longtest may have been fixed by https://go.dev/cl/516856.

If so, that may indicate that the openbsd/arm64 failure had a different root cause.
(attn @4a6f656c @golang/openbsd)

[aclements]

Yes, the two linux-arm64-longtest failures were fixed by https://go.dev/cl/516856.

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "openbsd" && goarch == "arm64" && log ~ `fatal error: traceback did not unwind completely`

2023-08-21 18:00 openbsd-arm64-jsing go@c7383441 os/signal.test [build] (log)

go.go(/home/gopher/build/tmp/go-link-3327794622/go.o:(syscall.libc_syscall_trampoline.abi0)): warning: syscall() may go away, please rewrite code to use direct calls

2023-08-21 18:00 openbsd-arm64-jsing go@c7383441 strings (log)

runtime: g1: frame.sp=0x400019fe00 top=0x40001bffd0
	stack=[0x4000180000-0x40001c0000
fatal error: traceback did not unwind completely

runtime stack:
runtime.throw({0x171b13?, 0x40001bffb0?})
	/home/gopher/build/go/src/runtime/panic.go:1018 +0x40 fp=0x4564aba10 sp=0x4564ab9e0 pc=0x452a0
runtime.(*unwinder).finishInternal(0x202f0?)
	/home/gopher/build/go/src/runtime/traceback.go:561 +0x110 fp=0x4564aba50 sp=0x4564aba10 pc=0x6b7a0
runtime.(*unwinder).next(0x4564abb28?)
	/home/gopher/build/go/src/runtime/traceback.go:442 +0x2a4 fp=0x4564abae0 sp=0x4564aba50 pc=0x6b5e4
runtime.scanstack(0x40000021a0, 0x10?)
	/home/gopher/build/go/src/runtime/mgcmark.go:802 +0x23c fp=0x4564abf20 sp=0x4564abae0 pc=0x2d8dc
runtime.markroot.func1()
	/home/gopher/build/go/src/runtime/mgcmark.go:240 +0xb4 fp=0x4564abf70 sp=0x4564abf20 pc=0x2c664
runtime.markroot(0x400002a140, 0x14, 0x1)
	/home/gopher/build/go/src/runtime/mgcmark.go:214 +0x1cc fp=0x4564ac020 sp=0x4564abf70 pc=0x2c32c
runtime.gcDrain(0x400002a140, 0x7)
	/home/gopher/build/go/src/runtime/mgcmark.go:1069 +0x3f8 fp=0x4564ac080 sp=0x4564ac020 pc=0x2e308
runtime.gcBgMarkWorker.func2()
	/home/gopher/build/go/src/runtime/mgc.go:1405 +0x74 fp=0x4564ac0d0 sp=0x4564ac080 pc=0x2a504
runtime.systemstack()
	/home/gopher/build/go/src/runtime/asm_arm64.s:243 +0x6c fp=0x4564ac0e0 sp=0x4564ac0d0 pc=0x765bc

— watchflakes

[yihuang]

not sure if it's the same root cause, but I got similar error message with this package and go 1.21:

prysmaticlabs/gohashtree#13

[potuz]

not sure if it's the same root cause, but I got similar error message with this package and go 1.21:

prysmaticlabs/gohashtree#13

This issue is reproducible on amd64 so doesn't seem to be arm64 related.