Found by the lab proof of #406 (the image-pinning move, 2026-09-12); corrects the "skip path" statement of that design. Fixed in v4.8.1 (#410).
Problem
Since 4.8.0 the kagent Harness is rendered by the kagent chart (harness.create: true), owned by the kagent HelmRelease; the connectivity chart no longer renders it (4.7.19 added helm.sh/resource-policy: keep so the in-order upgrade hands it over without a gap). The design assumed that a Harness deleted by a consumer skipping 4.7.19 (an exactly pinned connectivity chart without keep) "is recreated by the kagent release one reconcile later". It is not:
kubectl -n kagent delete harness kagent at 16:34:08Z, then reconcile.fluxcd.io/requestedAt on the kagent HelmRelease: helm-controller logged "all dependencies are ready … release in-sync with desired state" and ran no helm upgrade. The kagent HelmRelease has no spec.driftDetection (default disabled); status.history[0] stayed at the previous version.- 600 s observed with the Harness absent; both admitted
AgentTemplates had an empty .status.harnesses and were not runnable for the whole window.
reconcile.fluxcd.io/forceAt (+ requestedAt) on the kagent HelmRelease recreated it in 1 s (new uid), templates Ready +23 s, first turn +26 s.
Consumers whose connectivity chart follows the range already carry 4.7.19's keep and take the in-order path implicitly; the exposure is an exactly pinned connectivity chart jumping 4.7.18 → 4.8.0.
Root cause
helm-controller's reconcile of a HelmRelease whose chart and values are unchanged compares the desired release (chart version + values) with the stored release and reports in-sync; the cluster state is only compared when spec.driftDetection.mode is enabled (or warn). Without it a deleted object of the release is invisible to the controller until the next values or chart change or a forced reconcile.
Solution (option 1, shipped in v4.8.1 — #410)
spec.driftDetection.mode: enabled on the kagent HelmRelease. On every reconcile (gitops.interval, 10 min by default) helm-controller compares the manifest in the Helm storage with the cluster and re-applies what differs or is missing, as a server-side apply — no Helm revision, no forced reconcile.
components.<name>.driftDetection is a passthrough for every component entry (HelmRelease v2 spec.driftDetection: mode, ignore), set for kagent only; turning it on for another release is a decision per release (its objects must tolerate the re-apply). The kagent chart (0.11.0-gs.6) renders no hook, webhook or HPA-scaled Deployment and nothing edits its objects in place, so no ignore rule is needed. The Harness carries keep itself and is the release's own object — not drift.tests/verify-kagent-wiring.py::check_drift_detection asserts the kagent release carries it and no other release does by default (in make verify-meta).- The ATS smoke
test_deleted_platform_harness_comes_back_on_the_next_reconcile deletes the Harness, requests a plain reconcile (requestedAt, standing in for the interval — no forceAt), and asserts it is back with a new uid owned by the kagent release with keep, spec unchanged, both templates Ready again, Helm revision unchanged.
- UPGRADE.md: a
<current> → <next> section for the change; the 4.8.0 skip-path bullet now states the truth for an installation on 4.8.0 exactly and names the forced-reconcile recipe (kubectl -n <gitops.namespace> annotate helmrelease kagent reconcile.fluxcd.io/requestedAt=$(date -u +%Y-%m-%dT%H:%M:%SZ) reconcile.fluxcd.io/forceAt=$(date -u +%Y-%m-%dT%H:%M:%SZ) --overwrite).
examples/customer-bom.yaml pins the connectivity chart to the release it is published with (a pin behind 4.7.19 renders a second owner of the Harness next to the kagent release).
Option 2 (the UPGRADE note alone) was rejected as the only answer: it makes the runtime of every agent depend on an operator knowing a recipe on a path they may not know they are on. It stays as the note for installations on 4.8.0 exactly.
Acceptance criteria
Found by the lab proof of #406 (the image-pinning move, 2026-09-12); corrects the "skip path" statement of that design. Fixed in v4.8.1 (#410).
Problem
Since 4.8.0 the
kagentHarness is rendered by the kagent chart (harness.create: true), owned by thekagentHelmRelease; the connectivity chart no longer renders it (4.7.19 addedhelm.sh/resource-policy: keepso the in-order upgrade hands it over without a gap). The design assumed that a Harness deleted by a consumer skipping 4.7.19 (an exactly pinned connectivity chart withoutkeep) "is recreated by the kagent release one reconcile later". It is not:kubectl -n kagent delete harness kagentat 16:34:08Z, thenreconcile.fluxcd.io/requestedAton the kagent HelmRelease: helm-controller logged "all dependencies are ready … release in-sync with desired state" and ran nohelm upgrade. The kagent HelmRelease has nospec.driftDetection(defaultdisabled);status.history[0]stayed at the previous version.AgentTemplates had an empty.status.harnessesand were not runnable for the whole window.reconcile.fluxcd.io/forceAt(+requestedAt) on the kagent HelmRelease recreated it in 1 s (new uid), templates Ready +23 s, first turn +26 s.Consumers whose connectivity chart follows the range already carry 4.7.19's
keepand take the in-order path implicitly; the exposure is an exactly pinned connectivity chart jumping 4.7.18 → 4.8.0.Root cause
helm-controller's reconcile of a HelmRelease whose chart and values are unchanged compares the desired release (chart version + values) with the stored release and reports in-sync; the cluster state is only compared when
spec.driftDetection.modeisenabled(orwarn). Without it a deleted object of the release is invisible to the controller until the next values or chart change or a forced reconcile.Solution (option 1, shipped in v4.8.1 — #410)
spec.driftDetection.mode: enabledon the kagent HelmRelease. On every reconcile (gitops.interval, 10 min by default) helm-controller compares the manifest in the Helm storage with the cluster and re-applies what differs or is missing, as a server-side apply — no Helm revision, no forced reconcile.components.<name>.driftDetectionis a passthrough for every component entry (HelmRelease v2spec.driftDetection:mode,ignore), set for kagent only; turning it on for another release is a decision per release (its objects must tolerate the re-apply). The kagent chart (0.11.0-gs.6) renders no hook, webhook or HPA-scaled Deployment and nothing edits its objects in place, so no ignore rule is needed. The Harness carrieskeepitself and is the release's own object — not drift.tests/verify-kagent-wiring.py::check_drift_detectionasserts the kagent release carries it and no other release does by default (inmake verify-meta).test_deleted_platform_harness_comes_back_on_the_next_reconciledeletes the Harness, requests a plain reconcile (requestedAt, standing in for the interval — noforceAt), and asserts it is back with a new uid owned by the kagent release withkeep, spec unchanged, both templates Ready again, Helm revision unchanged.<current> → <next>section for the change; the 4.8.0 skip-path bullet now states the truth for an installation on 4.8.0 exactly and names the forced-reconcile recipe (kubectl -n <gitops.namespace> annotate helmrelease kagent reconcile.fluxcd.io/requestedAt=$(date -u +%Y-%m-%dT%H:%M:%SZ) reconcile.fluxcd.io/forceAt=$(date -u +%Y-%m-%dT%H:%M:%SZ) --overwrite).examples/customer-bom.yamlpins the connectivity chart to the release it is published with (a pin behind 4.7.19 renders a second owner of the Harness next to the kagent release).Option 2 (the UPGRADE note alone) was rejected as the only answer: it makes the runtime of every agent depend on an operator knowing a recipe on a path they may not know they are on. It stays as the note for installations on 4.8.0 exactly.
Acceptance criteria
tests/verify-kagent-wiring.pyasserts it — fix(kagent): the kagent release detects and corrects drift — a deleted platform Harness comes back on the next reconcile (#409) #410 (squash b917fbd, released as v4.8.1;push-chart-releaseandpush-connectivity-chart-releasegreen on the tag;helm show values oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.8.1carriescomponents.kagent.driftDetection.mode: enabledand its render putsspec.driftDetection.mode: enabledon the kagent HelmRelease). The UPGRADE notes carry the corrected skip-path statement with the forced-reconcile recipe for 4.8.0 exactly as well.platform.chartPath, helm-controller v1.6.4 / Flux 2.9.5): the first reconcile with drift detection reportedDrifted=False NoDriftDetected(no spurious drift on the kagent chart's objects).kubectl -n kagent delete harness kagentat 18:29:55Z with no annotation of any kind: both templates unadmitted at +10 s; the next interval reconcile at 18:38:58Z loggedDriftDetected: Harness/kagent/kagent removed→DriftCorrected: Harness/kagent/kagent created; the Harness back at +545 s (new uid,meta.helm.sh/release-name: kagent,keep); both templates Ready at +567 s; Helm revision 11 unchanged.platform-test9/9 andagents-test6/6 (a turn as the person on the recreated Harness) afterwards; the lab restored to the released chart. On the PR the ATS smoke (execute-chart-tests, required) ran the new test green.