kagent.substrateWorkerPool: one pool must be one CPU feature set, not just one architecture — mixed AMD/Intel Karpenter nodes wedge actor restores (incompatible FeatureSet) #429

Problem

kagent.substrateWorkerPool.template.nodeSelector pins the pool to kubernetes.io/arch: amd64 and the values comment says "one pool runs one CPU architecture: an actor's golden snapshot is architecture-specific". The real constraint is narrower: a golden snapshot is a gVisor checkpoint, and gVisor restores only on a host whose CPU offers every feature the checkpoint recorded. Two amd64 nodes of different vendors are not compatible.

Observed on graveler, 2026-09-13 (meta chart 4.9.4, kagent 0.11.0-gs.8, Substrate 0.0.27-gs.8), giantswarm/giantswarm#37745 portal proof: the Karpenter pool holds AMD c5a.xlarge and Intel r6i.xlarge / m7i-flex.xlarge nodes. Three of the four kagent-default workers landed on c5a, one on r6i. The sre-agent golden snapshot was taken on an AMD worker. Every restore of the AgentInstance the scheduler placed on the Intel worker failed, and the turn surfaced as actor "ai-…" request timed out after 60 s in the portal:

Actor restoring
FATAL ERROR: starting container: starting sub-container [/app]: failed to load kernel:
incompatible FeatureSet: missing features: map[cmp_legacy:{} cr8_legacy:{} sse4a:{} misalignsse:{} topoext:{} mmxext:{} fxsr_opt:{}]

(all seven are AMD-only CPUID flags). The worker retried the restore every 30 s with the same result; the instance never recovers because the actor stays placed on that worker.

Root cause

The chart and the fleet template pinned the pool by architecture only (kubernetes.io/arch: amd64), and every Giant Swarm CAPA installation on the 4.x line provisions its worker nodes through Karpenter, whose consolidation replaces a node with any instance type the NodePool admits — graveler's admits every non-burstable family with 4–32 vCPUs on Nitro, AMD and Intel alike. gVisor restores a checkpoint only where the host CPU offers every feature it recorded, so the golden snapshot taken on an AMD worker could not load on the Intel one, and Substrate retries the restore on the same claimed worker (validateAssignedWorker finds it live, owned and eligible — the scheduler's constraints know nothing about the CPU).

Two things sharpen the constraint beyond "one vendor": a snapshot taken on a newer generation of one vendor does not restore on an older one either (a c7a snapshot on m5a), so the strictest pin is an instance family; and the Karpenter label exists only where Karpenter provisions the nodes — of the CAPA management clusters only garm, gazelle, graveler and grizzly run it, so a fleet-wide pin needs an opt-out for the others.

Fix (merged 2026-09-13)

• docs(meta): the WorkerPool is one CPU feature set, not one architecture — vendor and family pins on CAPA, node-pool pin elsewhere, recognition recipe (#429) #430 — the values comment, the chart README, the README's "Scheduling" paragraph and UPGRADE.md name the feature-set constraint (one vendor and generation, not just one architecture), the failure signature, the pin per provider (CAPA: karpenter.k8s.aws/instance-cpu-manufacturer: amd|intel, karpenter.k8s.aws/instance-family where a generation change must be ruled out; CAPZ and others: the node pool giantswarm.io/machine-pool or node.kubernetes.io/instance-type) and the recognition recipe. The knob stays the free-form template.nodeSelector map: the right key is provider- and autoscaler-specific, which the fleet template knows and the chart does not, so no dedicated values key and no render-time guard.
• giantswarm/shared-configs#735 — agentPlatform.workerPoolCpuManufacturer: amd in default/config.yaml; the 4.x block renders karpenter.k8s.aws/instance-cpu-manufacturer: <vendor> into the pool's nodeSelector for every capa installation; empty renders no pin (a CAPA installation without Karpenter labels). amd is the vendor Karpenter picks for the fleet's pools and the one the existing golden snapshots were taken on.
• giantswarm/giantswarm-configs#828 — graveler's per-installation bridge (fix(connectivity): kagent controller cannot reach atenet-router #827) removed; the rendered ConfigMap is byte-identical before and after.
• docs(fork): upstream item — place a restore only on a worker whose CPU covers the snapshot's feature set (#37742 row 12) substrate#29 — the durable fix is Substrate's: the upstream-facing proposal (workers report their CPU feature set, snapshots record it, Constraints.RequiredCPUFeatures as a superset check with a RESUMING actor rescheduled instead of retried in place, and a fast-fail classification of the incompatible FeatureSet restore error) is prepared in the line's FORK.md and tracked as giantswarm/giantswarm#37742 row 12, to file after the team's review (fork-first rule).

Verification

• Render: graveler's agent-platform values rendered locally with konfigure three ways — the old template with the bridge, the new template with the bridge, the new template without it — are byte-identical (the key is rendered once, from the template).
• Live on graveler after both merges (Konfiguration Applied revision: e92b3670…): WorkerPool kagent-default spec.template.nodeSelector = {"karpenter.k8s.aws/instance-cpu-manufacturer":"amd","kubernetes.io/arch":"amd64"}; the four kagent-default workers are the pods created at 12:58–12:59Z (no re-roll), on ip-10-0-125-162 and ip-10-0-246-28, both c5a.xlarge / amd. The portal turn that failed passes since the pool was pinned (giantswarm/giantswarm#37745, session 3).
• Not visible in agentlab: one kind node, one CPU — a mixed pool cannot occur there. The chart change is documentation only (helm lint, pre-commit clean).

Acceptance criteria

• helm/agent-platform/values.yaml (substrateWorkerPool.template.nodeSelector comment) and the connectivity/kagent docs name the feature-set constraint and the vendor label.
• shared-configs' fleet template pins karpenter.k8s.aws/instance-cpu-manufacturer for CAPA installations (or the chart guards a mixed pool).
• agentlab's Kyverno/Cilium-less lab is not the place to see this; the trial report documents the recognition recipe (worker log incompatible FeatureSet, node instance-cpu-manufacturer labels).