Skip to content

fix(substrate): re-pin the line to v0.0.27-gs.3 — the ate.dev CRDs are kept on uninstall, so a cluster's own Flux uninstalls the platform without racing substrate-crds - #390

Merged
teemow merged 1 commit into
mainfrom
fix-385-substrate-crds-keep
Sep 11, 2026
Merged

teemow merged 1 commit into
mainfrom
fix-385-substrate-crds-keep

Conversation

@teemow

@teemow teemow commented Sep 11, 2026

Copy link
Copy Markdown
Member

What

Closes #385 — the decision recorded there: helm.sh/resource-policy: keep on the Substrate line's CRD templates, released as v0.0.27-gs.3 (giantswarm/substrate#12, tag on 92eec3c6, publish run 34603612441) and re-pinned here.

  • The pins (make verify-components holds them to one): components.substrate / components.substrate-crds >=0.0.27-gs.3 <0.0.28-0, kagent.substrateWorkerPool.workerImage ghcr.io/giantswarm/substrate/ateom-gvisor:0.0.27-gs.3, examples/customer-bom.yaml, tests/verify-components.py, the chart README (helm-docs).
  • The ATS own-Flux scenario's way back is a test (test_the_way_back_uninstalls_through_the_clusters_flux, last in the module): the agent's HelmRelease deleted first, then the meta HelmRelease; the cluster's Flux uninstalls the seven platform releases concurrently — the test aborts on any UninstallFailed / failed to delete release condition — and asserts the eight kept CRDs are in place with the annotation, no SandboxConfig or WorkerPool outlives the release that owned it, muster, Substrate's control plane and the kagent workloads are gone. delete_platform_releases_in_waves is removed; the fixture teardown only removes what the test did not reach and the Flux install.
  • The smoke's ordered teardown asserts the same keep policy on the three ate.dev CRDs (assert_kept_crds, shared), the SandboxConfig gone with the substrate release and no WorkerPool left, instead of "Substrate's CRDs gone".
  • Docs: README "Clusters that run Flux" (a new "Uninstalling through that Flux" paragraph: the order, why it is safe, what stays, the pre-gs.3 caveat), "Uninstalling" (the kept ate.dev CRDs), "Agent Substrate" and "CRD lifecycle" (substrate-crds is keep-annotated now — the row was the one exception to the section's promise), UPGRADE.md (a <current> → <next> entry: no operator action to install, the changed uninstall semantics, the order on own-Flux clusters), tests/ats/README.md, CHANGELOG.md.

Why

Uninstalling the platform through a cluster's own Flux (engine off — every management cluster) deletes every component HelmRelease at once and helm-controller finalizes them concurrently; substrate-crds could go before substrate, whose uninstall then failed for good on its SandboxConfig (failed to delete release: substrate: Helm cannot delete an object whose kind is gone) and was retried until the meta HelmRelease's finalizer timed out — measured by #380 (CircleCI job 5941), worked around in the test since. With the CRDs kept the order stops mattering: substrate deletes its SandboxConfig, kagent its WorkerPool, substrate-crds leaves the CRDs. Neither ate-controller nor the kagent controller sets a finalizer on the CRs, so nothing else in a concurrent uninstall waits for a controller that is gone.

Rejected (on #385): rendering the teardown-releases hook with the engine off (a cluster-admin hook in the fleet render, against the verify-engine contract, for an order that stops mattering) and waiting for a Flux-side reverse-dependsOn uninstall (upstream's open fluxcd/flux2#1744).

Verification

No installation acts — the release stays inert behind the fleet's <4.0.0 bound.

@circleci-architect

circleci-architect Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform

4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c
Chart agent-platform
Version 4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c
Digest sha256:212089149e5dbfe5dea8d302f0b48d255ac262318ba709494d98f1acea847962
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c

Posted by architect-orb · build 6147 · commit 2b1201c · updated in place on every push

@circleci-architect

circleci-architect Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Published Helm chart agent-platform-connectivity

4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c
Chart agent-platform-connectivity
Version 4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c
OCI reference oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c
Digest sha256:eb2206dcb864cf83ef7aa57ea06cfe3960701b2e2fe1629c150e039cfe36c865
Registry public — gsoci.azurecr.io
Git catalog giantswarm-test-catalog (index)
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.7.7-dev.fix-385-sub--e-crds-keep.2026-09-11.13-47-51.h2b1201c

Posted by architect-orb · build 6149 · commit 2b1201c · updated in place on every push

…e kept on uninstall, so a cluster's own Flux uninstalls the platform without racing substrate-crds

Uninstalling the platform through a cluster's own Flux (the fleet's path,
engine off) deleted every component HelmRelease at once; helm-controller
finalized them concurrently and substrate-crds could go before substrate,
whose uninstall then failed for good on its SandboxConfig (Helm cannot delete
an object whose kind is gone) and was retried until the meta HelmRelease's
finalizer timed out (#385, measured on the ATS own-Flux scenario).

The Substrate line's v0.0.27-gs.3 carries helm.sh/resource-policy: keep on the
three ate.dev CRD templates (giantswarm/substrate#12, the kagent-crds
convention), so a consumer's uninstall can always delete its CRs whatever
order the releases go in. The two versionRanges, the BOM pin and the
worker-image tag move to gs.3 together.

The own-Flux scenario's way back is a test: the meta HelmRelease deleted, the
seven releases uninstalled concurrently with no UninstallFailed, the eight
kept CRDs in place, no SandboxConfig or WorkerPool left; the wave-deletion
workaround is gone. The smoke's ordered teardown asserts the keep policy on
ate.dev too. README "Clusters that run Flux", "Uninstalling", "CRD lifecycle"
and UPGRADE.md describe the uninstall order and what a kept CRD implies.

Closes #385
@teemow
teemow force-pushed the fix-385-substrate-crds-keep branch from fa10ce9 to 2b1201c Compare September 11, 2026 13:47
@teemow
teemow merged commit 6460fed into main Sep 11, 2026
13 checks passed
@teemow
teemow deleted the fix-385-substrate-crds-keep branch September 11, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant