kagent.substrateWorkerPool: resilience of the one worker pool — PodDisruptionBudget, karpenter.sh/do-not-disrupt and karpenter.sh/capacity-type: on-demand as documented and verified knobs, topologySpreadConstraints forwarded behind a render guard once the Substrate line accepts them #472

Problem

The platform runs every agent on one Substrate WorkerPool (kagent.substrateWorkerPool, 4 gVisor workers, one actor each). On a Karpenter installation the pool is only as robust as the node its workers land on:

• After the CPU-generation pin (kagent WorkerPool spans AMD generations (m7a and c5ad): golden snapshots taken on gen 7 cannot restore on gen 5 workers, half of the default agent's cold turns fail after 60 s #457) Karpenter bin-packed all four workers of gazelle's kagent-default onto one spot m6a.xlarge (2026-09-15, ip-10-0-145-23) although two other generation-6 nodes existed. A spot reclaim or a consolidation of that node takes the whole pool down: every turn in flight fails, every session waiting for a person is lost (its pause checkpoint is node-local, giantswarm/giantswarm#37795), and new turns fail no free workers available until replacements schedule.
• Nothing guards the workers against voluntary disruption: no PodDisruptionBudget selects them (the kagent namespace carries budgets for kagent-controller and the database only), no karpenter.sh/do-not-disrupt annotation is set, so an hourly WhenEmptyOrUnderutilized consolidation may drain a worker node whenever a cheaper packing exists.
• Every worker node in gazelle is spot; Substrate's own upgrade runbook (docs/upgrade.md) presumes no spot or preemptible nodes under workers and that a serving pool is never edited casually.
• WorkerPool.spec.template (Substrate v0.0.30-gs.1, pkg/api/v1alpha1/workerpool_types.go) takes labels, annotations, nodeSelector, tolerations, priorityClassName, nodeAffinity, resources — not topologySpreadConstraints or podAntiAffinity. The CRD is a structural schema without preserve-unknown-fields, so a spread constraint set through kagent.substrateWorkerPool.template today is pruned silently by the apiserver: the values look applied and do nothing. The kagent chart forwards the map verbatim and has no values schema.

Why not more pools instead: kagent pins every ActorTemplate to the one pool of the Harness that admitted it (kagent.dev/worker-pool selector), the client names the Harness a session runs on (Swarmgeist takes the first Ready Harness, the Dev Portal the labelled one, agent-manager composes one global name), and one (AgentTemplate, Harness) pair is one golden snapshot — a second pool is a second Harness, doubled golden boots and idle capacity nobody routes to. The scheduler places on labels and randomness only. The robust layout is one homogeneous pool per Harness, spread and guarded.

Decision (Timo, 2026-09-15)

The worker pool stays on spot and the workers get a PodDisruptionBudget only — no karpenter.sh/do-not-disrupt by default. Both knobs (annotation, capacity-type selector) stay documented and forwarded for an installation that wants them; nothing sets them on gazelle. The spread half proceeds now on the Substrate line (giantswarm/giantswarm#37797).

Proposed solution

Values and templates in the meta chart; the Substrate half (the CRD fields) is a carried patch on the Substrate line, tracked as giantswarm/giantswarm#37797 (giantswarm/giantswarm#37742 row 46); the fleet template's knobs are giantswarm/shared-configs#738.

1. PodDisruptionBudget for the worker pods, rendered by the connectivity chart the way muster-valkey's budget is (feat(meta): the stateful singletons can be pinned to on-demand capacity (scheduling.singletons, opt-in); muster-valkey takes the #431 guards (#439) #442): kagent.substrateWorkerPool.podDisruptionBudget: {enabled: true, maxUnavailable: 1} (default on), selector ate.dev/worker-pool: <substrateWorkerPool.name>, in the kagent namespace. Voluntary drains then move one worker at a time.
2. Document and verify the two knobs that work today in values.yaml, both READMEs and UPGRADE.md: template.annotations: {karpenter.sh/do-not-disrupt: "true"} (consolidation never drains a worker node; a drift roll waits for the NodePool's terminationGracePeriod) and template.nodeSelector: {karpenter.sh/capacity-type: on-demand} (no involuntary reclaim; the only knob that covers a spot interruption). Note the roll: every spec.template change rolls the worker Deployment once (RollingUpdate 25 %/25 %, one worker at a time), turns in flight on a replaced worker are lost, goldens are untouched.
3. Forward template.topologySpreadConstraints (and podAntiAffinity) behind a render guard: agent-platform.validateWorkerPool fails the render naming the key while components.substrate.versionRange's floor is below the Substrate release that carries the fields, so a value that the apiserver would prune silently becomes a failed render; forwarded verbatim from that release on. Recommended values once available: hostname maxSkew 1, minDomains 2, whenUnsatisfiable DoNotSchedule (Karpenter provisions the second node) and zone maxSkew 1, ScheduleAnyway, labelSelector on ate.dev/worker-pool: <name>.
4. make verify-workerpool extended: the PDB renders with the pool's selector and maxUnavailable: 1; the annotation and the capacity-type selector reach the kagent release verbatim; a spread value fails the render below the floor and reaches the WorkerPool above it; still exactly one WorkerPool.
5. controller.substrate.defaultWorkerPool is inert on API v2 (the kagent chart renders SUBSTRATE_DEFAULT_WORKERPOOL_*, nothing in the line reads them) — say so in values.yaml rather than presenting it as a knob.

Status (2026-09-15)

Implemented in #473, merged 2026-09-15 14:07Z → released as agent-platform 4.20.0 (the chart half). What the chart does now:

• kagent.substrateWorkerPool.podDisruptionBudget (enabled: true, maxUnavailable: 1, unhealthyPodEvictionPolicy: AlwaysAllow) renders PodDisruptionBudget <pool> in the kagent namespace from the connectivity chart (templates/kagent/workerpool-pdb.yaml), selecting ate.dev/worker-pool: <pool>; never forwarded to the kagent release (components.kagent.omitKeys). The connectivity chart's four budgets share one spec helper (agent-platform.podDisruptionBudget.spec).
• template.annotations karpenter.sh/do-not-disrupt and template.nodeSelector karpenter.sh/capacity-type: on-demand documented (values.yaml, both READMEs, UPGRADE.md), unset by default, forwarded verbatim; the roll every spec.template change causes documented.
• agent-platform.validateWorkerPool refuses topologySpreadConstraints / podAntiAffinity while components.substrate.versionRange's floor is below agent-platform.substrate.workerPoolSpreadFloor (a one-line define in _helpers.tpl, empty until the Substrate release of giantswarm/giantswarm#37797 is out — refused unconditionally today, correct because every published release prunes them), and any other key WorkerPool.spec.template does not have. Follow-up — actionable now: the Substrate release is out, v0.0.30-gs.2 (comment below, 14:05Z; feat(kagent): the Substrate worker pool gets a PodDisruptionBudget; the pool template's Karpenter knobs are documented and verified; a spread value is refused until the Substrate line carries it #473 was in CI, so 4.20.0 shipped with the floor empty). The chart holds the Substrate floor and the kagent chart's stamped worker image to one version, so the order is: (1) giantswarm/kagent-upstream Makefile SUBSTRATE_VERSION 0.0.30-gs.1 → 0.0.30-gs.2 → the tag workflow stamps ateom-gvisor:0.0.30-gs.2 into 0.11.0-gs.19 (gs.1 → gs.2 changes no ateom code — ateapi, atecontroller, atenet and the CRD only); (2) here: agent-platform.substrate.workerPoolSpreadFloor → 0.0.30-gs.2, components.substrate / substrate-crds versionRange → >=0.0.30-gs.2 <0.0.31-0, components.kagent / kagent-crds → >=0.11.0-gs.19 <0.11.1-0, the BOM and tests/verify-components.py pins, UPGRADE.md (every installation rolls the Substrate control plane and the WorkerPool once and retakes every golden); make verify-workerpool then exercises the floor-set branch by itself; (3) the spread values in giantswarm/shared-configs#738, gated on that meta release.
• kagent.controller.substrate.defaultWorkerPool marked inert on API v2.
• make verify-workerpool asserts the budget (from the connectivity chart of the working tree), the two knobs on the kagent release and the resolved kagent chart's WorkerPool, the refused keys (the floor-set branch exercised with a temporary floor), one WorkerPool.

Lab evidence (shared agentlab, platform.chartPath + the branch's connectivity dev channel; details on the PR): the budget present with ALLOWED DISRUPTIONS 1 on four Ready workers and nothing rolled (WorkerPool spec, Deployment revision, AgentTemplate conditions unchanged); a second concurrent eviction refused with TooManyRequests: Cannot evict pod as it would violate the pod's disruption budget and ALLOWED DISRUPTIONS 0 while the first worker terminated; platform-test 9/9, agents-test 6/6, klaus-gateway-test 6/6; with a faked karpenter.sh/capacity-type=on-demand node label and both knobs set the WorkerPool.spec.template carries them verbatim, the Deployment rolled once (revision 3 → 4, one worker at a time under the budget), the pods carry the annotation and the selector, the AgentTemplate kept its conditions (no re-snapshot).

gazelle (verified 2026-09-15 14:20Z): Flux rolled agent-platform and agent-platform-connectivity to 4.20.0 at 14:20:37Z (release v124, both Ready). kubectl -n kagent get pdb now lists kagent-default N/A 1 1 next to kagent-controller, kagent-pg and kagent-pg-primary; its status currentHealthy: 4, desiredHealthy: 3, disruptionsAllowed: 1, DisruptionAllowed=True (SufficientPods), the selector ate.dev/worker-pool: kagent-default. The ten AgentTemplates are identical to the pre-release capture (generation and Ready transition times unchanged — no re-snapshot); the WorkerPool.spec.template is the fleet's three-label pin plus resources, untouched by the roll. The four workers run on two on-demand c6a.xlarge generation-6 nodes (the fleet's on-demand selector was reverted at 14:09Z, giantswarm-configs#841 — that revert, not 4.20.0, was the worker roll seen at 14:09Z; Karpenter still placed the replacements on on-demand nodes). The remaining gazelle item (four workers on at least two nodes by the spread, twenty cold threads, zero incompatible FeatureSet / dispatch error) belongs to the spread follow-up: the Substrate release of giantswarm/giantswarm#37797, then the floor here and the values in giantswarm/shared-configs#738.

Acceptance criteria

• With default values the kagent namespace carries a PodDisruptionBudget selecting ate.dev/worker-pool: kagent-default with maxUnavailable: 1 (ALLOWED DISRUPTIONS 1 with four Ready workers); podDisruptionBudget.enabled: false removes it.
• values.yaml, both READMEs and UPGRADE.md document template.annotations (karpenter.sh/do-not-disrupt) and template.nodeSelector karpenter.sh/capacity-type: on-demand with their effect, the Karpenter caveats (a do-not-disrupt pod blocks a drift drain without a NodePool terminationGracePeriod; the capacity-type label is Karpenter-only — never on CAPZ or on-prem) and the roll a template change causes.
• A template.topologySpreadConstraints value fails the render with a message naming the key and the Substrate floor while the pinned range lacks the field, and reaches WorkerPool.spec.template unchanged once the range's floor carries it; the same for podAntiAffinity.
• make verify-workerpool asserts all of the above and stays green in CI.
• Lab (agentlab, platform.chartPath): the PDB exists and refuses a second concurrent eviction, the annotation and selector are on the WorkerPool and the worker pods, platform-test, agents-test and klaus-gateway-test green; with a faked node label karpenter.sh/capacity-type=on-demand on the kind node the workers reschedule and the AgentTemplates keep their revisions.
• gazelle after the chart lands: the worker PDB exists in kagent (ALLOWED DISRUPTIONS 1), template revisions unchanged (no re-snapshot); once the spread ships (giantswarm/shared-configs#738 values), the four workers on at least two nodes, twenty consecutive new Slack threads on the default agent answered, zero incompatible FeatureSet, zero dispatch error.