runtime: fatal error: mcall called on m->g0 stack on Windows

[RichieSams]

Go version

go1.21.4

Output of go env in your module/workspace:

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2751320750=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We have a cli program that runs on many thousands of VMs / bare metal workers as part of our larger CI setup. Recently, we've been seeing a non-trivial number of panics on a specific set of workers. These workers are bare metal AMD Threadripper machines running Windows 10. Below is some info from DXDiag. (I can get additional information if needed).

      Time of this report: 4/23/2024, 09:36:37
             Machine name: XXXXXXXX
               Machine Id: {F429D65A-427C-4F90-A587-8AB68B33FB57}
         Operating System: Windows 10 Enterprise 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
                 Language: English (Regional Setting: English)
      System Manufacturer: Equus Computer Systems
             System Model: Nobilis
                     BIOS: F5c (type: UEFI)
                Processor: AMD Ryzen Threadripper 3990X 64-Core Processor  (64 CPUs), ~2.9GHz
                   Memory: 262144MB RAM
      Available OS Memory: 262032MB RAM
                Page File: 38200MB used, 668832MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.0.8
           DxDiag Version: 10.00.19041.3636 64bit Unicode

We're seeing two types of panics:

1. fatal error: runtime: mcall called on m->g0 stack
2. init order seemingly not following the spec: https://go.dev/ref/spec#Package_initialization
   • init() from a package is being called before the init() of a package it depends on
   • Meaning a global variable is not yet populated
   • Leading to a nil pointer dereference panic

I have included multiple call stacks of these panics below. NOTE: these callstacks all happen on different machines. But they only happen on the same "series" of machine, which is documented above.

All the panics seem to happen during the init() phase of the runtime start-up. The panics are not super reproduce-able. I'm only really seeing it due to our scale.

What did you see happen?

Below are the callstacks from a number of panics across multiple machines.

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1801948?, 0x2f?})
	runtime/panic.go:1077 +0x65 fp=0xc00025bc20 sp=0xc00025bbf0 pc=0x60c7a5
runtime.badmcall(0xd0?)
	runtime/proc.go:503 +0x1f fp=0xc00025bc40 sp=0xc00025bc20 pc=0x60fa9f
runtime.badmcall(0x60db19)
	<autogenerated>:1 +0x25 fp=0xc00025bc58 sp=0xc00025bc40 pc=0x641ea5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00025bc70
stack: frame={sp:0xc00025bc40, fp:0xc00025bc58} stack=[0x6fc804000,0x6fc9ff7b0)
goroutine 1 [running, locked to thread]:
google.golang.org/genproto/googleapis/type/expr.init()
	google.golang.org/genproto@v0.0.0-20240227224415-6ceb2ff114de/googleapis/type/expr/expr.pb.go:149 +0x69 fp=0xc00025be10 sp=0xc00025be08 pc=0xc11629
runtime.doInit1(0x243c990)
	runtime/proc.go:6740 +0xdd fp=0xc00025bf40 sp=0xc00025be10 pc=0x61bf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc00025bfe0 sp=0xc00025bf40 pc=0x60f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00025bfe8 sp=0xc00025bfe0 pc=0x63d241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x60f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x63d241
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x5f8ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x5edc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x1a65f48?, 0x1?, 0x0?, 0xc0002041a0?)
	runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x60f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2768440)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x5f6369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x5f68fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x5edbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5
goroutine 19 [finalizer wait]:
runtime.gopark(0x400000?, 0x1000f9e70?, 0x0?, 0x0?, 0x27cba60?)
	runtime/proc.go:398 +0xce fp=0xc0000f9e28 sp=0xc0000f9e08 pc=0x60f40e
runtime.runfinq()
	runtime/mfinal.go:193 +0x107 fp=0xc0000f9fe0 sp=0xc0000f9e28 pc=0x5ecce7
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x63d241
created by runtime.createfing in goroutine 1
	runtime/mfinal.go:163 +0x3d

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1522440?, 0x20?})
	/usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc00083f748 sp=0xc00083f718 pc=0x2bc045
runtime.badmcall(0xa?)
	/usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc00083f768 sp=0xc00083f748 pc=0x2bf2c5
runtime.badmcall(0x0)
	<autogenerated>:1 +0x2c fp=0xc00083f780 sp=0xc00083f768 pc=0x2ec86c
runtime.asyncPreempt2()
	/usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc00083f7a0 sp=0xc00083f780 pc=0x2bd27f
runtime.asyncPreempt()
	/usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc00083f928 sp=0xc00083f7a0 pc=0x2ebbbb
github.com/dustin/go-humanize.revfmap(0xc00024f680)
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36ac0
github.com/dustin/go-humanize.init()
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
	/usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
	/usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 1 [runnable, locked to thread]:
github.com/dustin/go-humanize.revfmap(0xc00024f680?)
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 +0xf0 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36bb0
github.com/dustin/go-humanize.init()
	/builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
	/usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
	/usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
	/usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f3fb0 sp=0xc0000f3f90 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
	/usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc0000f3fe0 sp=0xc0000f3fb0 pc=0x2be9b1
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f3fe8 sp=0xc0000f3fe0 pc=0x2ea561
created by runtime.init.6
	/usr/local/go/src/runtime/proc.go:290 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010ff90 sp=0xc00010ff70 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
	/usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00010ffc8 sp=0xc00010ff90 pc=0x2a7c37
runtime.gcenable.func1()
	/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x29c786
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x2ea561
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:178 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc000106000?, 0x179b290?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111f70 sp=0xc000111f50 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x2432600)
	/usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000111fa0 sp=0xc000111f70 pc=0x2a5c73
runtime.bgscavenge(0x0?)
	/usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000111fc8 sp=0xc000111fa0 pc=0x2a6285
runtime.gcenable.func2()
	/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x29c726
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x2ea561
created by runtime.gcenable
	/usr/local/go/src/runtime/mgc.go:179 +0xaa
goroutine 34 [finalizer wait]:
runtime.gopark(0x0?, 0xc0000f7e70?, 0x6b?, 0x7d?, 0xc0000f7f70?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f7e28 sp=0xc0000f7e08 pc=0x2beb16
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
	/usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc0000f7fe0 sp=0xc0000f7e28 pc=0x29b88f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x2ea561
created by runtime.createfing
	/usr/local/go/src/runtime/mfinal.go:157 +0x45
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010bf50 sp=0xc00010bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010bfe0 sp=0xc00010bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 3 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f5f50 sp=0xc0000f5f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000f5fe0 sp=0xc0000f5f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011df50 sp=0xc00011df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011dfe0 sp=0xc00011df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011ff50 sp=0xc00011ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011ffe0 sp=0xc00011ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010df50 sp=0xc00010df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010dfe0 sp=0xc00010df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000119f50 sp=0xc000119f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000119fe0 sp=0xc000119f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 4 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000505f50 sp=0xc000505f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000505fe0 sp=0xc000505f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000507f50 sp=0xc000507f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000507fe0 sp=0xc000507f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000501f50 sp=0xc000501f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000501fe0 sp=0xc000501f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000127f50 sp=0xc000127f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000127fe0 sp=0xc000127f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000127fe8 sp=0xc000127fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000129f50 sp=0xc000129f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000129fe0 sp=0xc000129f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000129fe8 sp=0xc000129fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011bf50 sp=0xc00011bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011bfe0 sp=0xc00011bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000503f50 sp=0xc000503f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000503fe0 sp=0xc000503f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000123f50 sp=0xc000123f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000123fe0 sp=0xc000123f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000125f50 sp=0xc000125f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000125fe0 sp=0xc000125f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050df50 sp=0xc00050df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050dfe0 sp=0xc00050df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 26 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000509f50 sp=0xc000509f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000509fe0 sp=0xc000509f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050ff50 sp=0xc00050ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050ffe0 sp=0xc00050ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 27 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050bf50 sp=0xc00050bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050bfe0 sp=0xc00050bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 10 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000515f50 sp=0xc000515f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000515fe0 sp=0xc000515f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 28 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000511f50 sp=0xc000511f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000511fe0 sp=0xc000511f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000517f50 sp=0xc000517f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000517fe0 sp=0xc000517f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000517fe8 sp=0xc000517fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 29 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000513f50 sp=0xc000513f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000513fe0 sp=0xc000513f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000513fe8 sp=0xc000513fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 12 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051df50 sp=0xc00051df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051dfe0 sp=0xc00051df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 30 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000519f50 sp=0xc000519f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000519fe0 sp=0xc000519f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000519fe8 sp=0xc000519fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 13 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051ff50 sp=0xc00051ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051ffe0 sp=0xc00051ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051ffe8 sp=0xc00051ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 31 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051bf50 sp=0xc00051bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051bfe0 sp=0xc00051bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051bfe8 sp=0xc00051bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 14 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000525f50 sp=0xc000525f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000525fe0 sp=0xc000525f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000525fe8 sp=0xc000525fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000521f50 sp=0xc000521f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000521fe0 sp=0xc000521f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000521fe8 sp=0xc000521fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000527f50 sp=0xc000527f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000527fe0 sp=0xc000527f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000527fe8 sp=0xc000527fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 40 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000523f50 sp=0xc000523f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000523fe0 sp=0xc000523f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000523fe8 sp=0xc000523fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 16 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052ff50 sp=0xc00052ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052ffe0 sp=0xc00052ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 41 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052bf50 sp=0xc00052bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052bfe0 sp=0xc00052bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052bfe8 sp=0xc00052bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000531f50 sp=0xc000531f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000531fe0 sp=0xc000531f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 42 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052df50 sp=0xc00052df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052dfe0 sp=0xc00052df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000537f50 sp=0xc000537f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000537fe0 sp=0xc000537f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000537fe8 sp=0xc000537fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 43 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533f50 sp=0xc000533f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000533fe0 sp=0xc000533f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000533fe8 sp=0xc000533fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000539f50 sp=0xc000539f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000539fe0 sp=0xc000539f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 44 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000535f50 sp=0xc000535f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000535fe0 sp=0xc000535f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053ff50 sp=0xc00053ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 45 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053bf50 sp=0xc00053bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053bfe0 sp=0xc00053bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000541f50 sp=0xc000541f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000541fe0 sp=0xc000541f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053df50 sp=0xc00053df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053dfe0 sp=0xc00053df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000547f50 sp=0xc000547f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000547fe0 sp=0xc000547f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 47 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000543f50 sp=0xc000543f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000543fe0 sp=0xc000543f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000549f50 sp=0xc000549f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000549fe0 sp=0xc000549f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 48 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000545f50 sp=0xc000545f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000545fe0 sp=0xc000545f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000545fe8 sp=0xc000545fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054ff50 sp=0xc00054ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054ffe0 sp=0xc00054ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054ffe8 sp=0xc00054ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 49 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054bf50 sp=0xc00054bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054bfe0 sp=0xc00054bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054bfe8 sp=0xc00054bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 58 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000551f50 sp=0xc000551f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000551fe0 sp=0xc000551f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000551fe8 sp=0xc000551fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 66 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054df50 sp=0xc00054df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054dfe0 sp=0xc00054df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 59 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000557f50 sp=0xc000557f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000557fe0 sp=0xc000557f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000557fe8 sp=0xc000557fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 67 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000553f50 sp=0xc000553f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000553fe0 sp=0xc000553f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000553fe8 sp=0xc000553fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 60 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000559f50 sp=0xc000559f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000559fe0 sp=0xc000559f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000559fe8 sp=0xc000559fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 68 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000555f50 sp=0xc000555f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000555fe0 sp=0xc000555f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000555fe8 sp=0xc000555fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 61 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055ff50 sp=0xc00055ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055ffe0 sp=0xc00055ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055ffe8 sp=0xc00055ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 69 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055bf50 sp=0xc00055bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055bfe0 sp=0xc00055bf50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055bfe8 sp=0xc00055bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 62 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000561f50 sp=0xc000561f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000561fe0 sp=0xc000561f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000561fe8 sp=0xc000561fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 70 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055df50 sp=0xc00055df30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055dfe0 sp=0xc00055df50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055dfe8 sp=0xc00055dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 63 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000567f50 sp=0xc000567f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000567fe0 sp=0xc000567f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 71 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000563f50 sp=0xc000563f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000563fe0 sp=0xc000563f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000563fe8 sp=0xc000563fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 64 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000569f50 sp=0xc000569f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000569fe0 sp=0xc000569f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 72 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000565f50 sp=0xc000565f30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000565fe0 sp=0xc000565f50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 65 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00056ff50 sp=0xc00056ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
	/usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00056ffe0 sp=0xc00056ff50 pc=0x29e991
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00056ffe8 sp=0xc00056ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
	/usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 83 [select]:
runtime.gopark(0xc000571f88?, 0x3?, 0xc8?, 0xc?, 0xc000571f72?)
	/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000571df8 sp=0xc000571dd8 pc=0x2beb16
runtime.selectgo(0xc000571f88, 0xc000571f6c, 0xc0001d2500?, 0x0, 0x0?, 0x1)
	/usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000571f38 sp=0xc000571df8 pc=0x2ccebc
go.opencensus.io/stats/view.(*worker).start(0xc0001d2500)
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad fp=0xc000571fc8 sp=0xc000571f38 pc=0x977c2d
go.opencensus.io/stats/view.init.0.func1()
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x976e66
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x2ea561
created by go.opencensus.io/stats/view.init.0
	/builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

Then here are some callstacks from init's that are seemingly not following the right init order:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x5 pc=0xb84adc]
goroutine 1 [running]:
golang.org/x/text/internal/language.addTags({0x139, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
	golang.org/x/text@v0.14.0/internal/language/match.go:99 +0x27c
golang.org/x/text/internal/language.Tag.Maximize(...)
	golang.org/x/text@v0.14.0/internal/language/match.go:68
golang.org/x/text/language.init.0()
	golang.org/x/text@v0.14.0/language/match.go:727 +0x26d

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x9 pc=0x132a0c0]
goroutine 1 [running]:
google.golang.org/protobuf/internal/filedesc.(*Enums).Len(0x3023620?)
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/desc_list_gen.go:25
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x23b2710, 0xc0003fc000}, 0xc00035f288)
	/builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:408 +0x5d
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc0000081f8, {0x23b2710?, 0xc0003fc000?})
	/builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:149 +0x739
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x239d908, ...}, ...})
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/build.go:112 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x0, ...}, ...}, ...})
	/builds/my-project/vendor/google.golang.org/protobuf/internal/filetype/build.go:138 +0x1b8
google.golang.org/genproto/googleapis/rpc/code.file_google_rpc_code_proto_init()
	/builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:331 +0x198
google.golang.org/genproto/googleapis/rpc/code.init.0()
	/builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:313 +0x17

What did you expect to see?

The program finishes the init() phase without panics

@gabyhelp's overview of this issue: #67108 (comment)

[RichieSams]

Below is my go.mod file (with a few redactions)

module my-project/v3

require (
	github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2
	github.com/antlr4-go/antlr/v4 v4.13.0
	github.com/bmatcuk/doublestar v1.3.4
	github.com/bsipos/thist v1.0.0
	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
	github.com/dustin/go-humanize v1.0.1
	github.com/getsentry/sentry-go v0.27.0
	github.com/go-ole/go-ole v1.3.0
	github.com/gobwas/glob v0.2.3
	github.com/gofrs/flock v0.8.1
	github.com/gonum/stat v0.0.0-20181125101827-41a0da705a5b
	github.com/gorilla/handlers v1.5.2
	github.com/gorilla/mux v1.8.1
	github.com/hashicorp/go-multierror v1.1.1
	github.com/jotfs/fastcdc-go v0.2.0
	github.com/json-iterator/go v1.1.12
	github.com/juju/ratelimit v1.0.2
	github.com/karrick/godirwalk v1.16.2
	github.com/olekukonko/tablewriter v0.0.5
	github.com/pkg/errors v0.9.1
	github.com/restic/chunker v0.4.0
	github.com/satori/go.uuid v1.2.0
	github.com/schollz/progressbar/v3 v3.14.2
	github.com/sirupsen/logrus v1.9.3
	github.com/skratchdot/open-golang v0.0.0-20190104022628-a2dfa6d0dab6
	github.com/spf13/cobra v1.8.0
	github.com/spf13/viper v1.18.2
	github.com/stretchr/testify v1.9.0
	github.com/thediveo/enumflag v0.10.1
	github.com/tinylib/msgp v1.1.9
	github.com/xhit/go-str2duration/v2 v2.1.0
	internal-a v1.7.0
	internal-b v1.2.0
	internal-c v1.7.0
	internal-d/v2 v2.70.0
	go.opentelemetry.io/otel v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.25.0
	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.25.0
	go.opentelemetry.io/otel/metric v1.25.0
	go.opentelemetry.io/otel/sdk v1.25.0
	go.opentelemetry.io/otel/sdk/metric v1.25.0
	go.opentelemetry.io/otel/trace v1.25.0
	golang.org/x/sync v0.7.0
	golang.org/x/sys v0.19.0
	golang.org/x/term v0.19.0
	golang.org/x/text v0.14.0
	google.golang.org/grpc v1.63.2
	gopkg.in/yaml.v2 v2.4.0
)

require (
	cloud.google.com/go v0.112.0 // indirect
	cloud.google.com/go/compute v1.24.0 // indirect
	cloud.google.com/go/compute/metadata v0.2.3 // indirect
	cloud.google.com/go/iam v1.1.6 // indirect
	cloud.google.com/go/storage v1.36.0 // indirect
	github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
	github.com/Microsoft/go-winio v0.6.0 // indirect
	github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
	github.com/Shopify/toxiproxy/v2 v2.4.0 // indirect
	github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af // indirect
	github.com/aws/aws-sdk-go v1.28.9 // indirect
	github.com/beorn7/perks v1.0.1 // indirect
	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/containerd/continuity v0.3.0 // indirect
	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/docker/cli v20.10.17+incompatible // indirect
	github.com/docker/docker v20.10.7+incompatible // indirect
	github.com/docker/go-connections v0.4.0 // indirect
	github.com/docker/go-units v0.4.0 // indirect
	github.com/fatih/color v1.14.1 // indirect
	github.com/felixge/httpsnoop v1.0.4 // indirect
	github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90 // indirect
	github.com/fsnotify/fsnotify v1.7.0 // indirect
	github.com/go-logr/logr v1.4.1 // indirect
	github.com/go-logr/stdr v1.2.2 // indirect
	github.com/gogo/protobuf v1.3.2 // indirect
	github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
	github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.4 // indirect
	github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac // indirect
	github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 // indirect
	github.com/gonum/integrate v0.0.0-20181209220457-a422b5c0fdf2 // indirect
	github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 // indirect
	github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 // indirect
	github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 // indirect
	github.com/google/s2a-go v0.1.7 // indirect
	github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
	github.com/google/uuid v1.6.0 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
	github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
	github.com/hashicorp/errwrap v1.0.0 // indirect
	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
	github.com/hashicorp/go-hclog v1.5.0 // indirect
	github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
	github.com/hashicorp/hcl v1.0.0 // indirect
	github.com/imdario/mergo v0.3.12 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect
	github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5 // indirect
	github.com/kylelemons/godebug v1.1.0 // indirect
	github.com/magiconair/properties v1.8.7 // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/mattn/go-runewidth v0.0.14 // indirect
	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
	github.com/minio/minio-go/v6 v6.0.46 // indirect
	github.com/minio/sha256-simd v0.1.1 // indirect
	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
	github.com/mitchellh/go-homedir v1.1.0 // indirect
	github.com/mitchellh/mapstructure v1.5.0 // indirect
	github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/opencontainers/go-digest v1.0.0 // indirect
	github.com/opencontainers/image-spec v1.0.2 // indirect
	github.com/opencontainers/runc v1.1.5 // indirect
	github.com/ory/dockertest/v3 v3.10.0 // indirect
	github.com/pelletier/go-toml/v2 v2.1.0 // indirect
	github.com/philhofer/fwd v1.1.2 // indirect
	github.com/pierrec/lz4 v2.0.5+incompatible // indirect
	github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
	github.com/prometheus/client_golang v1.12.1 // indirect
	github.com/prometheus/client_model v0.2.0 // indirect
	github.com/prometheus/common v0.32.1 // indirect
	github.com/prometheus/procfs v0.7.3 // indirect
	github.com/rivo/uniseg v0.4.7 // indirect
	github.com/russross/blackfriday/v2 v2.1.0 // indirect
	github.com/sagikazarmark/locafero v0.4.0 // indirect
	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
	github.com/sourcegraph/conc v0.3.0 // indirect
	github.com/spf13/afero v1.11.0 // indirect
	github.com/spf13/cast v1.6.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/subosito/gotenv v1.6.0 // indirect
	github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.30.0 // indirect
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
	go.opentelemetry.io/proto/otlp v1.1.0 // indirect
	go.uber.org/atomic v1.9.0 // indirect
	go.uber.org/multierr v1.9.0 // indirect
	golang.org/x/crypto v0.21.0 // indirect
	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
	golang.org/x/image v0.0.0-20190802002840-cff245a6509b // indirect
	golang.org/x/mod v0.13.0 // indirect
	golang.org/x/net v0.23.0 // indirect
	golang.org/x/oauth2 v0.17.0 // indirect
	golang.org/x/time v0.5.0 // indirect
	golang.org/x/tools v0.14.0 // indirect
	gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b // indirect
	google.golang.org/api v0.162.0 // indirect
	google.golang.org/appengine v1.6.8 // indirect
	google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda // indirect
	google.golang.org/protobuf v1.33.0 // indirect
	gopkg.in/ini.v1 v1.67.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	nhooyr.io/websocket v1.8.10 // indirect
)

replace github.com/antlr/antlr4 v0.0.0-20180905200841-432022fc1ca0 => github.com/michilu/antlr4 v0.0.0-20180803091604-411960b1878f

replace github.com/uber/jaeger-lib v2.0.0+incompatible => github.com/uber/jaeger-lib v1.5.0

replace github.com/skratchdot/open-golang => github.com/seanhoughton/open-golang v0.0.0-20190322203133-7972e18a724d

replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0

go 1.21

[RichieSams]

This seems to be related to: #56774

But in that issue, the poster said updating to 1.21.3 fixed it. However, we're using 1.21.4, and still seeing the issue.

[RichieSams]

Slightly different panic:

unexpected fault address 0x411a89e
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x411a89e pc=0xe003e1]
goroutine 1 [running, locked to thread]:
runtime.throw({0x1f23e6f?, 0x0?})
	runtime/panic.go:1077 +0x65 fp=0xc0000fbd58 sp=0xc0000fbd28 pc=0xd9c7a5
runtime.sigpanic()
	runtime/signal_windows.go:377 +0xd0 fp=0xc0000fbda0 sp=0xc0000fbd58 pc=0xdb06b0
time.map.init.1()
	time/zoneinfo_abbrs_windows.go:154 +0x81 fp=0xc0000fbe00 sp=0xc0000fbda0 pc=0xe003e1
time.init()
	time/zoneinfo_abbrs_windows.go:15 +0x25 fp=0xc0000fbe10 sp=0xc0000fbe00 pc=0xe001a5
runtime.doInit1(0x2b81dc0)
	runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0xdabf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0xd9f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0xdcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0xd9f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xdcd141
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0xd88ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0xd7dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x21b8428?, 0x1?, 0x0?, 0xc0000f4b60?)
	runtime/proc.go:398 +0xce fp=0xc000109f70 sp=0xc000109f50 pc=0xd9f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x2e63e20)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc000109fa0 sp=0xc000109f70 pc=0xd86369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc000109fc8 sp=0xc000109fa0 pc=0xd868fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc000109fe0 sp=0xc000109fc8 pc=0xd7dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc000109fe8 sp=0xc000109fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

[RichieSams]

A new type of panic:

fatal error: runtime: name offset out of range
goroutine 1 [running, locked to thread]:
runtime.throw({0x187d65c?, 0x1850bb5?})
	runtime/panic.go:1077 +0x65 fp=0xc0000fbcf8 sp=0xc0000fbcc8 pc=0x68c7a5
runtime.resolveNameOff(0x284d14a?, 0x6c5160)
	runtime/type.go:119 +0x245 fp=0xc0000fbd50 sp=0xc0000fbcf8 pc=0x6b1765
internal/reflectlite.resolveNameOff(0x0?, 0x0?)
	runtime/runtime1.go:626 +0x13 fp=0xc0000fbd70 sp=0xc0000fbd50 pc=0x6b82d3
internal/reflectlite.rtype.nameOff(...)
	internal/reflectlite/type.go:244
internal/reflectlite.rtype.String({0xc000218030?})
	internal/reflectlite/type.go:256 +0x1b fp=0xc0000fbd98 sp=0xc0000fbd70 pc=0x6c4d7b
internal/reflectlite.elem(0xc0000fbdf0?)
	internal/reflectlite/type.go:320 +0x73 fp=0xc0000fbdd0 sp=0xc0000fbd98 pc=0x6c5113
internal/reflectlite.rtype.Elem({0xc000218000?})
	internal/reflectlite/type.go:324 +0x13 fp=0xc0000fbde8 sp=0xc0000fbdd0 pc=0x6c5173
errors.init()
	errors/wrap.go:136 +0x43 fp=0xc0000fbe10 sp=0xc0000fbde8 pc=0x6c90c3
runtime.doInit1(0x24bccc0)
	runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0x69bf1d
runtime.doInit(...)
	runtime/proc.go:6707
runtime.main()
	runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0x68f096
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0x6bd241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.forcegchelper()
	runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0x68f298
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x6bd241
created by runtime.init.6 in goroutine 1
	runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.bgsweep(0x0?)
	runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0x678ab4
runtime.gcenable.func1()
	runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0x66dc25
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x1ae7600?, 0x1?, 0x0?, 0xc0002041a0?)
	runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x68f40e
runtime.goparkunlock(...)
	runtime/proc.go:404
runtime.(*scavengerState).park(0x27ea440)
	runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x676369
runtime.bgscavenge(0x0?)
	runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x6768fc
runtime.gcenable.func2()
	runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x66dbc5
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
	runtime/mgc.go:201 +0xa5

[cherrymui]

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

These workers are bare metal AMD Threadripper machines running Windows 10.

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines? Do you see it only on a specific type of Windows machines? Thanks.

[RichieSams]

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines?

Correct. We cross compile in docker. But we're running in native Windows.

Do you see it only on a specific type of Windows machines? Thanks.

Yes. We only see it on our AMD servers. Example specs of the machines is listed above in the original post. None of our Intel-based servers are seeing these panics

Let me know if you have any other questions. :)

[cherrymui]

Thanks. Have you tried running the program under the race detector? Is the program using cgo? Is there a way we can reproduce the issue ourselves?