containerd crash when pausing a gvisor container

[yangchi]

Description

I think there is a regression introduced between release-20260105.0 and release-20260112.0. The latter is the first version I can reproduce the problem. I think it affects all versions since then (latest at the moment is release-20260216.0)

Or maybe something introduced between these two versions makes it require certain version of containerd ?

Our containerd version v2.1.4

We use gVisor in our kubernetes environment with containerd. Recently I noticed if I pause a running gVisor container through containerd, containerd crashes with this stack:

panic: protobuf tag not enough fields in Status.state:
goroutine 97 [running]:
github.com/gogo/protobuf/proto.(*unmarshalInfo).computeUnmarshalInfo(0xc0002121e0)
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/table_unmarshal.go:341 +0x127f
github.com/gogo/protobuf/proto.(*unmarshalInfo).unmarshal(0xc0002121e0, {0xc00026cea0?}, {0xc000225803, 0x98, 0x16e6})
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/table_unmarshal.go:138 +0x5b
github.com/gogo/protobuf/proto.typeUnmarshaler.makeUnmarshalMessagePtr.func51({0xc000225801, 0x9a, 0x16e8}, {0xc00027bbf8?}, 0x43b3d7?)
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/table_unmarshal.go:1826 +0x145
github.com/gogo/protobuf/proto.(*unmarshalInfo).unmarshal(0xc000212140, {0x8d2140?}, {0xc000225800?, 0x20?, 0x8f60a0?})
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/table_unmarshal.go:175 +0x34a
github.com/gogo/protobuf/proto.(*InternalMessageInfo).Unmarshal(0xc0001a8a00?, {0x9f6c48, 0xc0001a89a0}, {0xc000225800?, 0x9b?, 0x16e9?})
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/table_unmarshal.go:63 +0xb0
github.com/gogo/protobuf/proto.(*Buffer).Unmarshal(0xc000091e18, {0x9f6c48, 0xc0001a89a0})
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/decode.go:424 +0x165
github.com/gogo/protobuf/proto.Unmarshal({0xc000225800, 0x9b, 0x16e9}, {0x9f6c48, 0xc0001a89a0})
        external/gazelle++go_deps+com_github_gogo_protobuf/proto/decode.go:342 +0x10a
github.com/containerd/ttrpc.(*Client).recv(0xc00006d140?, 0xd?, 0xc000171fd0?)
        external/gazelle++go_deps+com_github_containerd_ttrpc/client.go:378 +0xe8
github.com/containerd/ttrpc.(*Client).run.func2()
        external/gazelle++go_deps+com_github_containerd_ttrpc/client.go:318 +0x205
created by github.com/containerd/ttrpc.(*Client).run in goroutine 62
        external/gazelle++go_deps+com_github_containerd_ttrpc/client.go:286 +0x13f

Also this log may be related too, right before the panic:

Feb 25 03:46:50 containerd[2944]: time="2026-02-25T03:46:50.749004975Z" level=info msg="no topic for type &events.TaskPaused{ContainerID:\"b74324c10a794af8f0cd351c66a38e6eedec0f7d220015a6d6a6f3fd83ef30d0\", XXX_NoUnkeyedLiteral:struct {}{}, XX>

This happen with both using containerd client or just simply ctr -n k8s.io tasks pause TASK_ID command. It reproduce reliable in our environment. I have not tested if this also happens if I only use gVisor and containerd, without Kubernetes.

Steps to reproduce

(1) Start a gVisor container via kubernetes, with containerd
(2) Either with containerd client, or with ctr command on the host, pause it the container with its task ID

ctr -n k8s.io tasks pause TASK_ID

(you may need to enter namespace of PID 1 if this is a k8s host)

(3) Now the pod will be in Error state. ctr -n k8s.io tasks list will no longer have this task. And containerd log will have the crash stack (journalctr -xeu containerd.service if you use systemd)

runsc version

from release-20260112.0 to current latest (release-20260216.0)

docker version (if using docker)

uname

Linux ip-10-173-130-206.us-east-2.compute.internal 6.12.46-66.121.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Sep 22 16:35:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

kubectl (if using Kubernetes)

Client Version: v1.34.4
Kustomize Version: v5.7.1
Server Version: v1.34.3-eks-ac2d5a0

repo state (if built from source)

No response

runsc debug logs (if available)

[milantracy]

Any chance you can try with containerd 2.1.0.

We have some test coverage with 2.1.0, I just want to make sure that it is not a bug from upstream containerd

[yangchi]

i think maybe tomorrow or day after I can find some time to test with containerd 2.1.0

[milantracy]

i don't think gvisor exsiting tests cover puase command, i will add some coverage to see if i can reproduce.