context: cancelCtx.Done panics because atomic.Value holds an interface{} instead of a chan struct{}

[shaj13]

Go version

go version go1.24.0 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2123548078=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

After deploying our pods to the staging environment, we observed a panic originating from the Go context package approximately 7 days later. The panic was triggered by a call to cancelCtx.Done, where the internal atomic.Value unexpectedly held an interface{} that was not a chan struct{}. This behavior is ambiguous, as cancelCtx.done is expected to always hold a channel.

Our application creates a chain of contexts: it begins with the gRPC request context (which carries gRPC tags), then wraps it with a new context that includes a logger, and later adds tracing information, The context may evolve with additional layers or sub-contexts depending on the call stack.

What did you see happen?

2025-05-23T03:42:04.170059178Z stderr F panic: interface conversion: interface {} is , not chan struct {}
2025-05-23T03:42:04.170084549Z stderr F 
2025-05-23T03:42:04.170094852Z stderr F goroutine 97982046 gp=0xc004b4ee00 m=18 mp=0xc000601008 [running]:
2025-05-23T03:42:04.173399855Z stderr F panic({0x20064c0?, 0xc001cc6570?})
2025-05-23T03:42:04.173437135Z stderr F 	/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.1.linux-amd64/src/runtime/panic.go:811 +0x168 fp=0xc000aaf350 sp=0xc000aaf2a0 pc=0x479528
2025-05-23T03:42:04.173447406Z stderr F runtime.panicdottypeE(0x3a00960, 0x1ebb780, 0x1f91640)
2025-05-23T03:42:04.173487512Z stderr F 	/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.1.linux-amd64/src/runtime/iface.go:275 +0x65 fp=0xc000aaf370 sp=0xc000aaf350 pc=0x415dc5
2025-05-23T03:42:04.173498219Z stderr F context.(*cancelCtx).Done(0xc000aaf448?)
2025-05-23T03:42:04.173506744Z stderr F 	/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.1.linux-amd64/src/context/context.go:445 +0x1a5 fp=0xc000aaf3e8 sp=0xc000aaf370 pc=0x52f165
2025-05-23T03:42:04.173515209Z stderr F context.(*valueCtx).Done(0x7496a393a948?)
2025-05-23T03:42:04.173524257Z stderr F 	<autogenerated>:1 +0x24 fp=0xc000aaf400 sp=0xc000aaf3e8 pc=0x531f64
2025-05-23T03:42:04.173532769Z stderr F context.(*valueCtx).Done(0x700000000601008?)
2025-05-23T03:42:04.173541216Z stderr F 	<autogenerated>:1 +0x24 fp=0xc000aaf418 sp=0xc000aaf400 pc=0x531f64
2025-05-23T03:42:04.173561515Z stderr F context.(*valueCtx).Done(0x50?)
2025-05-23T03:42:04.173570249Z stderr F 	<autogenerated>:1 +0x24 fp=0xc000aaf430 sp=0xc000aaf418 pc=0x531f64
2025-05-23T03:42:04.173580021Z stderr F context.(*cancelCtx).propagateCancel(0xc001cb4fa0, {0x269ad30, 0xc001cc6420}, {0x2680778, 0xc001cb4fa0})
2025-05-23T03:42:04.173588538Z stderr F 	/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.1.linux-amd64/src/context/context.go:469 +0x75 fp=0xc000aaf4b0 sp=0xc000aaf430 pc=0x52f335
2025-05-23T03:42:04.173597405Z stderr F context.withCancel(...)
2025-05-23T03:42:04.173606206Z stderr F 	/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.1.linux-amd64/src/context/context.go:278
2025-05-23T03:42:04.173614923Z stderr F context.WithCancel({0x269ad30, 0xc001cc6420})
2025-05-23T03:42:04.173623673Z stderr F 	/root/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.1.linux-amd64/src/context/context.go:241 +0x59 fp=0xc000aaf4f0 sp=0xc000aaf4b0 pc=0x52e8d9
2025-05-23T03:42:04.173632886Z stderr F google.golang.org/grpc.newClientStreamWithParams({0x269ad30, 0xc001cc6420}, 0x39aef80, 0xc003c21808, {0x23bb28f, 0x2e}, {0x0, 0x0, 0x0, 0x0, ...}, ...)
2025-05-23T03:42:04.173641379Z stderr F 	/vendor/google.golang.org/grpc/stream.go:275 +0x15c fp=0xc000aaf668 sp=0xc000aaf4f0 pc=0x9ba2fc
2025-05-23T03:42:04.173649851Z stderr F google.golang.org/grpc.newClientStream.func3({0x269ad30?, 0xc001cc6420?}, 0xc001cc6420?)
2025-05-23T03:42:04.173658313Z stderr F 	/vendor/google.golang.org/grpc/stream.go:222 +0x87 fp=0xc000aaf6f8 sp=0xc000aaf668 pc=0x9ba0e7
2025-05-23T03:42:04.173665777Z stderr F google.golang.org/grpc.newClientStream({0x269ad30, 0xc001cc6420}, 0x39aef80, 0xc003c21808, {0x23bb28f, 0x2e}, {0xc0038042a0, 0x1, 0x1?})
2025-05-23T03:42:04.173672839Z stderr F 	/vendor/google.golang.org/grpc/stream.go:257 +0x764 fp=0xc000aaf860 sp=0xc000aaf6f8 pc=0x9b9ac4
2025-05-23T03:42:04.173679712Z stderr F google.golang.org/grpc.(*ClientConn).NewStream(0x479f8e?, {0x269ad30?, 0xc001cc6420?}, 0x0?, {0x23bb28f?, 0xc004b4ee00?}, {0x0?, 0x456369?, 0xc000aaf940?})
2025-05-23T03:42:04.173686584Z stderr F 	/vendor/google.golang.org/grpc/stream.go:172 +0x1a5 fp=0xc000aaf8d8 sp=0xc000aaf860 pc=0x9b92e5
2025-05-23T03:42:04.173702126Z stderr F /sdk/scrapepb.(*scraperClient).Scrape(0x4da9da?, {0x269ad30?, 0xc001cc6420?}, 0xc00070f800, {0x0?, 0xc000aafc48?, 0xc001cd6240?})

What did you expect to see?

cancelCtx.done is expected to always hold a channel.

[seankhliao]

it's unclear how you ended up in this state, but without a reproducer, I don't think there's much we can do.

[shaj13]

it's unclear how you ended up in this state, but without a reproducer, I don't think there's much we can do.

It’s not clear to me how a non-chan value got stored there in the context’s done field. What other details/info would help dig into this further ?

I only have the panic message available, and we first noticed this issue occurred about 2 days and 19 hours ago.

[gabyhelp]

Related Issues

• context: panic in parentCancelCtx when context timeout #64307 (closed)
• panic: interface conversion: interface {} is ^A, not chan struct {} #66400 (closed)
• cmd/vet: panic: interface conversion: types.Type is *types.Named, not *types.Signature #16143 (closed)
• context: panic: interface conversion: interface {} is bool, not chan struct {} // #51789 (closed)
• context: (*cancelCtx).Done() channel type assertion error #60681 (closed)
• affected/package: sync #53769 (closed)
• x/tools/gopls: panic in internal/golang.activeParameter #69552 (closed)
• cmd/compile, x/sync/errgroup: TestWithContext failing after devirtualization CL #42284 (closed)
• runtime: runtime crashes, reproducer same as #20427 #36371 (closed)

Related Discussions

• <-ctx.Done() panic

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

it's impossible under normal circumstances
https://cs.opensource.google/go/go/+/master:src/context/context.go;l=448;drc=3fd729b2a14a7efcf08465cbea60a74da5457f06;bpv=1;bpt=1?q=context&ss=go%2Fgo

I recommend looking to see if your code has other race conditions that may result in corrupted values

[adonovan]

It's worth running your tests under the race detector (-race), as data races may cause heap corruption, and heap corruption may manifest as a data race.

[Pandey123Amit]

*context.timerCtx {cancelCtx: context.cancelCtx {Context: (unreadable could not resolve interface type), mu: (sync.Mutex)(0xc00008d7e0), done: ("sync/atomic.Value")(0xc00008d7e8), children: map[context.canceler]struct {} [...], err: (unreadable could not resolve interface type), cause: (unreadable could not resolve interface type)}, timer: *time.Timer {C: (unreadable unreadable length: Invalid access to memory location.), initTimer: true}, deadline: time.Time(0001-07-05T19:21:36Z){wall: 21364376, ext: 16053696, loc: *(*time.Location)(0x14601f0)}}