Skip to content

fix: keep the single-replica components drainable (maxUnavailable 1) - #697

Merged
teemow merged 8 commits into
mainfrom
teemow/pdb-drain-safe
Sep 25, 2026
Merged

teemow merged 8 commits into
mainfrom
teemow/pdb-drain-safe

Conversation

@teemow

@teemow teemow commented Sep 25, 2026

Copy link
Copy Markdown
Member

Written by Timo's agent.

Problem

These PodDisruptionBudgets are minAvailable: 1, each on a single replica:

  • muster
  • klaus-gateway
  • agent-manager
  • muster-valkey
  • the kagent controller

That refuses every voluntary eviction. A cluster upgrade that rolls its machines drains each node, and the drain waits for its timeout on every node where one of these pods runs. The budgets came with #431 to keep Karpenter's consolidation off the streaming pods. But their karpenter.sh/do-not-disrupt annotation already covers that, and the budget additionally blocks every drain.

Change

Budgets: each is now maxUnavailable: 1. A drain evicts the single replica and goes on, with the usual restart gap. Karpenter's consolidation still stays away through the pods' do-not-disrupt annotation, which is unchanged.

Component Where the budget is set New shape
muster meta muster.podDisruptionBudget → muster chart minAvailable: "" (clears the chart's default of 1), maxUnavailable: 1
klaus-gateway meta klausGateway.podDisruptionBudget → klaus-gateway chart minAvailable: "", maxUnavailable: 1
kagent controller meta kagent.controller.pdb → kagent chart the override is dropped, so the chart's own default applies: maxUnavailable: 1
agent-manager agentManager.podDisruptionBudget (meta and connectivity) maxUnavailable: 1
muster-valkey valkey.podDisruptionBudget (meta and connectivity) maxUnavailable: 1

Platform Postgres: with postgres.instances: 1, the CNPG Cluster renders enablePDB: false. The operator's budget on a lone primary has no replica to switch over to and would hold a drain. With two or more instances (the default is three), the operator's budgets stay and it switches the primary over when its node drains.

Unchanged: vm-manager's minAvailable: 1. Evicting its pod ends every VM on the node, so it is a deliberate choice outside this change.

Test

  • make verify-disruption passes. It now asserts maxUnavailable: 1 on each budget and no minAvailable next to it, the guards (both fields, neither, a lone minAvailable passes through), and enablePDB: false at one Postgres instance only.
  • make verify-components-charts: every component chart renders the forwarded values, at the range resolution and at the BOM pin. muster, klaus-gateway and kagent render their budget as maxUnavailable: 1.
  • The helm-docs and schema hooks are regenerated and clean.

Checklist

  • Update changelog in CHANGELOG.md.
  • Make sure values.yaml and values.schema.json are valid.

The budgets of muster, klaus-gateway, agent-manager, muster-valkey and
the kagent controller were minAvailable: 1 on one replica, refusing
every eviction, so a cluster upgrade rolling its machines waited for
each drain's timeout. They are maxUnavailable: 1; Karpenter's
consolidation stays away through the pods' do-not-disrupt annotation
(#431), not the budget. A one-instance platform Postgres renders
enablePDB: false: the operator's primary budget has no replica to
switch over to. verify-disruption asserts all of it.
@teemow
teemow requested a review from a team as a code owner September 25, 2026 09:27
* origin/main:
  fix(netpol): admit the scrape of the kagent controller and atenet-egress ext-proc (#691)

# Conflicts:
#	CHANGELOG.md
verify-gpu-pool, verify-target and verify-wiring compare against origin/main, which still renders minAvailable: 1; they hold the budgets until GOLDEN_REF carries this change. The ATS drift test edits muster's maxUnavailable instead of minAvailable.
* origin/main:
  feat: one OTLP collector for every exporter (global.observability.traces.otlp) (#696)
Since #696 is on main, origin/main renders the kube-system otlp-gateway default too; the hold cut it from the head only, so verify-target failed on main and on every branch after it.
* origin/main:
  fix(substrate): keep Substrate's metrics on the scrape path (#694)

# Conflicts:
#	tests/verify-target.py
@circleci-architect

circleci-architect Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform

4.76.1-re950fef2t20260925110702hd55a40c
Chart agent-platform
Version 4.76.1-re950fef2t20260925110702hd55a40c
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.76.1-re950fef2t20260925110702hd55a40c
Digest sha256:461dc3b534ec3e4447e13576102a26549c0ae37ba9f7f211b0c4ba66ff87c150
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.76.1-re950fef2t20260925110702hd55a40c

Posted by architect-orb · build 14336 · commit d55a40c · updated in place on every push

@circleci-architect

circleci-architect Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.76.1-re950fef2t20260925110702hd55a40c
Chart agent-platform-connectivity
Version 4.76.1-re950fef2t20260925110702hd55a40c
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.76.1-re950fef2t20260925110702hd55a40c
Digest sha256:3282f527bd0673ce97586a70d7fea6fb4a8a01a72f2d58c1c9ab09e5ffbc391f
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.76.1-re950fef2t20260925110702hd55a40c

Posted by architect-orb · build 14338 · commit d55a40c · updated in place on every push

origin/main carries #694's key now; the hold cut it from the head only, so verify-target failed on main.
* origin/main:
  feat(model-serving): export the model pods' traces to the platform's OTLP gateway (#695)

# Conflicts:
#	tests/verify-target.py
@teemow
teemow merged commit 4fdb898 into main Sep 25, 2026
13 checks passed
@teemow
teemow deleted the teemow/pdb-drain-safe branch September 25, 2026 11:38
@teemow

teemow commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

From Timo's agent: released as v4.76.1, rolled onto a production installation, and every HelmRelease there is Ready.

  • The budgets of muster, klaus-gateway, agent-manager, muster-valkey and the kagent controller are maxUnavailable: 1, each with disruptionsAllowed 1 for its one pod.
  • A server-side dry-run drain of each node that runs one of those pods exits 0, evicts the pod, and no budget refuses it.
  • That installation's platform Postgres runs 3 instances, so the operator's budgets are kept, as intended.

teemow added a commit that referenced this pull request Sep 25, 2026
…en-606

* origin/main: (28 commits)
  feat(dashboards): turn done (final flush) p50 and p95 on Klaus Gateway (#701)
  feat: the managers', the portal's and mcp-kubernetes' traces to the platform's OTLP gateway (#700)
  feat(kagent): compact agent histories only past 600000 tokens (#699)
  fix: keep the single-replica components drainable (maxUnavailable 1) (#697)
  feat(model-serving): export the model pods' traces to the platform's OTLP gateway (#695)
  fix(substrate): keep Substrate's metrics on the scrape path (#694)
  feat: one OTLP collector for every exporter (global.observability.traces.otlp) (#696)
  fix(netpol): admit the scrape of the kagent controller and atenet-egress ext-proc (#691)
  fix(serving): make the models Gateway an internet-facing NLB on AWS (#692)
  fix(model-manager): reach the workload clusters' API servers (#689)
  feat(components): the CloudNativePG operator chart and image from gsoci (#671)
  fix(serving): tear the serving slice down in order, configs freed (#679)
  fix: derive the managers' OAuth login from muster's OAuth server (#678)
  feat(model-serving): stamp the cache claim's tier as annotations (#676)
  fix(substrate): the capz snapshot-store identity holds Storage Blob Delegator on the account (#686)
  fix(connectivity): the egress gateway reaches the Substrate credential provider (#685)
  feat(llm-routing): route the LLM endpoint by model, with API keys (#672)
  feat: the kagent line and the Substrate line at 1.1.0 (#665)
  fix(model-manager): read served models wherever model-manager serves (#681)
  feat(model-serving): check every shipped preset's parsers by model family (#674)
  ...

# Conflicts:
#	CHANGELOG.md
#	tests/verify-gpu-pool.py
teemow added a commit that referenced this pull request Sep 25, 2026
…reset-597

* origin/main: (28 commits)
  feat(dashboards): turn done (final flush) p50 and p95 on Klaus Gateway (#701)
  feat: the managers', the portal's and mcp-kubernetes' traces to the platform's OTLP gateway (#700)
  feat(kagent): compact agent histories only past 600000 tokens (#699)
  fix: keep the single-replica components drainable (maxUnavailable 1) (#697)
  feat(model-serving): export the model pods' traces to the platform's OTLP gateway (#695)
  fix(substrate): keep Substrate's metrics on the scrape path (#694)
  feat: one OTLP collector for every exporter (global.observability.traces.otlp) (#696)
  fix(netpol): admit the scrape of the kagent controller and atenet-egress ext-proc (#691)
  fix(serving): make the models Gateway an internet-facing NLB on AWS (#692)
  fix(model-manager): reach the workload clusters' API servers (#689)
  feat(components): the CloudNativePG operator chart and image from gsoci (#671)
  fix(serving): tear the serving slice down in order, configs freed (#679)
  fix: derive the managers' OAuth login from muster's OAuth server (#678)
  feat(model-serving): stamp the cache claim's tier as annotations (#676)
  fix(substrate): the capz snapshot-store identity holds Storage Blob Delegator on the account (#686)
  fix(connectivity): the egress gateway reaches the Substrate credential provider (#685)
  feat(llm-routing): route the LLM endpoint by model, with API keys (#672)
  feat: the kagent line and the Substrate line at 1.1.0 (#665)
  fix(model-manager): read served models wherever model-manager serves (#681)
  feat(model-serving): check every shipped preset's parsers by model family (#674)
  ...

# Conflicts:
#	CHANGELOG.md
#	tests/verify-gpu-pool.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant