teemow · Sep 12, 2026 · Sep 12, 2026 · Sep 12, 2026
@@ -13,6 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **A deleted platform Harness is recreated by the kagent release on its next reconcile** (giantswarm/agent-platform#409): the kagent HelmRelease carries `spec.driftDetection.mode: enabled` — on every reconcile (`gitops.interval`) helm-controller compares the release manifest with the cluster and re-applies what differs or is missing, as a server-side apply without a Helm revision. Without it a plain reconcile of the unchanged release reported "in-sync", and the Harness a consumer skipping 4.7.19 lost on the 4.8.0 upgrade stayed deleted — every template unadmitted, 600 s observed — until a forced reconcile. `components.<name>.driftDetection` is a passthrough for every component entry (HelmRelease v2 `spec.driftDetection`: `mode`, `ignore`), set for kagent only; `tests/verify-kagent-wiring.py` asserts it and that no other release carries one by default, and the ATS smoke deletes the Harness and sees it back on a requested (not forced) reconcile with both templates Ready and no revision written. UPGRADE.md's 4.8.0 note names the forced-reconcile recovery for an installation on 4.8.0 exactly; the customer BOM example pins the connectivity chart to the release it is published with.
 - **The substrate release pins the platform Harness's image in atelet's image cache, and the Substrate line re-pins to `v0.0.27-gs.6`** (giantswarm/giantswarm#37764; giantswarm/giantswarm#37742 rows 34 and 35): atelet's image GC evicted every image older than two minutes on every pass while the cache volume sat above the 85 % watermark — the Harness image included — so the first turn after a few idle minutes paid a cold pull and unpack that did not fit the resume's restore budget and failed with `context canceled`. The line's sixth release (giantswarm/substrate#21, #23, #24) makes the pinned set configurable (`--image-cache-pinned-images`; chart `atelet.imageCache.{gcPeriod,highPercent,lowPercent,minAge,maxBytes,pinnedImages}`, unset by default) and raises the router's request-parking budget to 30 s. The meta chart derives `atelet.imageCache.pinnedImages` for the `substrate` release from `kagent.harness.image` (`agent-platform.componentDerivedValues`, next to the CNPG connection Secret): the platform Harness's digest joins whatever `substrate.atelet.imageCache.pinnedImages` lists — the installation's own pins first, once each — so atelet pulls it at every GC pass and never evicts it, and a Harness re-pin moves the pinned set with it; no second copy of the digest anywhere. The other `imageCache` knobs stay the chart's defaults (an installation or a lab tunes them in `substrate.atelet.imageCache`). `components.substrate`/`components.substrate-crds` move to `>=0.0.27-gs.6 <0.0.28-0` and `kagent.substrateWorkerPool.workerImage` to `…:0.0.27-gs.6`; `examples/customer-bom.yaml` and `tests/verify-components.py` move with the pin. `make verify-components` asserts the derived pin (the default render, a re-pinned Harness image, an installation's own list). An installation on the 4.x line rolls the atelet DaemonSet once (`--image-cache-pinned-images=<Harness digest>`) on its next reconcile; the lab proof — no `Image cache evicting image record` for the Harness image, the first turn after ten idle minutes completing — is the audit's next step. **No installation acts** — the release stays inert behind the fleet's `<4.0.0` bound.
 - **The kagent line re-pins to `v0.11.0-gs.4`** (giantswarm/giantswarm#37763, giantswarm/giantswarm#37010): the line's fourth release (commit `ca399a3a`; the upstream pin stays kagent-dev/kagent `main` @ `6a6b54ff`) carries one more patch, giantswarm/kagent-upstream#15 — the A2A gateway fails a task whose dispatch to the AgentInstance runtime fails (a failing dial, a failing ingester start, or a runtime stream that errors before its first event) instead of leaving it `TASK_STATE_SUBMITTED` for good, which kept it as the instance's active task, refused every later message as a conflict and showed every client a turn that never ends; a submitted task the runtime does not know is failed by the reconciler after the one-minute dispatch grace period. `kagent.tag` moves to `0.11.0-gs.4` (the controller and UI image build) and `kagent.harness.image` to the `gs.4` Go ADK index digest `sha256:6e510b241e2d000b7d10879c9fde3dd34a37879854278f0a5d386426049a88bf` (meta and connectivity charts); `components.kagent`/`components.kagent-crds` keep the range `>=0.11.0-gs.1 <0.11.1-0`. `examples/customer-bom.yaml` and `tests/verify-kagent-crds.py` (`KAGENT_LINE_REF`) move with the pin. An installation on the 4.x line rolls the kagent controller, UI and the platform Harness to the new build on its next reconcile; the lab proof of the released chart is the audit's next step (giantswarm/giantswarm#37763). `make verify-wiring`'s whole-render comparison against `origin/main` leaves image references out, so a re-pin — which moves `kagent.harness.image` by design — no longer trips it.
 

@@ -311,6 +311,8 @@ Both register their MCP endpoint with muster through the chart's own `MCPServer`
 
 **What the chart forwards.** The `kagent:` block is upstream's chart shape (keys at the root): `registry: ghcr.io`, the fork's image names, `controller.substrate.*` (the controller talks to Substrate's `ate-api` and routes through `atenet` in `ate-system`), `substrateWorkerPool` (the `WorkerPool` `kagent-default` the platform Harness runs on: `replicas` bound the concurrently active agents, `template` carries resources and node placement; `workerImage` is the chart's stamp unless set), `harness` (the platform `Harness` `kagent` the chart renders — `create: true`, the snapshot location, `KAGENT_PROPAGATE_TOKEN`, the admission label `agent-platform.giantswarm.io/harness: kagent` with the chart's own `kagent.dev/harness` key nulled away; the image is the chart's stamped Go ADK digest unless `kagent.harness.image` is set), pgvector on the bundled Postgres (the line's memory API needs it; with the platform CNPG cluster set `postgres.vector.enabled`), `kmcp` / `kagent-tools` / `grafana-mcp` off. The connectivity chart's keys — `controllerRoute`, `uiRoute`, `modelConfigs`, `remoteMcpServers`, `serviceMonitor`, `oauth2ProxyIngress`, `fluxServiceAccountName` — stay in the block and are dropped on the way to the kagent release (`components.kagent.omitKeys`): the kagent chart has no schema and would carry them on silently. `kagent-crds:` keeps the chart's `kmcp` and `substrate` CRD subcharts off — muster fills kmcp's role, Substrate's CRDs come with the `substrate-crds` component.
 
+**Drift detection.** The kagent HelmRelease carries `spec.driftDetection.mode: enabled` (`components.kagent.driftDetection`; a passthrough every component entry takes, set for kagent only): on every reconcile — `gitops.interval` — helm-controller compares the release manifest with the cluster and re-applies what differs or is missing, as a server-side apply without a Helm revision. So the platform Harness — deleted by the 4.8.0 upgrade of a consumer whose pinned connectivity chart skipped 4.7.19's `keep`, and never recreated by a release without drift detection (a plain reconcile of an unchanged release reports "in-sync") — is back within one interval and every template it admits returns to Ready (giantswarm/agent-platform#409; the ATS smoke deletes it and sees it back). A hand edit to an object of the release is reverted the same way; an object opts out with the annotation `helm.toolkit.fluxcd.io/driftDetection: disabled`.
+
 **Prerequisites.** Kubernetes 1.35 with the three feature gates and Agent Substrate — the runtime the chart ships next to kagent; [Agent Substrate](#agent-substrate) has the details. Upgrading a 3.x installation: [UPGRADE.md](./UPGRADE.md).
 
 ### Agent Substrate
@@ -732,7 +734,7 @@ Two layers of checks, both in CI on every PR and both runnable locally.
 
 **Render assertions** (no cluster; `test-ingress-modes`, Helm 3.17.3): `make verify-modes verify-global verify-meta verify-engine verify-self verify-managers verify-kagent-netpol verify-kagent-discovery verify-kagent-crds verify-postgres verify-secrets verify-presets verify-components verify-components-charts verify-auto verify-identity verify-wiring verify-migration verify-kagent-storage-version` — `Makefile.custom.mk` describes each target. `verify-meta` also walks the two `values.schema.json` files against each other, nested keys included (`tests/verify-schema-symmetry.py`): every key the connectivity chart declares is settable through the meta chart, and every key the meta chart forwards is declared by the connectivity chart — a key missing on either side fails a render, at render time here or on the connectivity release on every installation. The three shapes of the chart map onto them: engine on + self on (the quick start) is `verify-self`'s; engine on + self off (the lab shape, the hand-back) is `verify-engine`'s and `verify-self`'s; engine off (the fleet: `--include-crds` renders no CRD, hook, operator or self object — the pure renderer) is `verify-engine`'s and `verify-meta`'s; the `auto` knobs — the vanilla shape without `--api-versions`, the fleet shape with the API groups passed — are `verify-auto`'s. `pre-commit run -a` regenerates the schemas and the helm-docs READMEs.
 
-**The ATS** (`execute-chart-tests`, a required check; [tests/ats/README.md](tests/ats/README.md)) runs two scenarios on one kind cluster on a 4-vCPU `large` CircleCI machine — a cluster created from [`.ats/kind-config.yaml`](.ats/kind-config.yaml) with the `ClusterTrustBundle`, `ClusterTrustBundleProjection` and `PodCertificateRequest` feature gates Agent Substrate needs: the **smoke** — the quick start on a bare cluster (the Gateway API CRDs, a lab Dex and an in-cluster registry the candidate is pushed to, together with the connectivity chart of the checkout, which the roster is pointed at through `components.agent-platform-connectivity.{repository,versionRange,insecure}`; `helm install --wait` with the bundled engine, muster with its OAuth server, dicebear, connectivity, kagent with its CRD chart, Substrate with its CRD chart, agent-manager, self-management on against that registry), then the adoption, the auth round trip (the 401 discovery chain, a Dex user through the password grant and through muster's login flow), Substrate from the chart under test (its WorkerPool's gVisor worker Ready, the platform Harness), the agent round trips on kagent API v2 (a declarative `AgentTemplate` Ready on the platform Harness; agent-manager's `create_agent` through muster as the Dex user → the chart `1.x` HelmRelease as `kagent-flux` → the `AgentTemplate` Ready on the Harness and the agent's `RemoteMCPServer` Accepted), the self-management fixpoint and values Secret, the refused `helm upgrade`, and the ordered teardown with the line's keep policy (the `kagent-crds` release uninstalled, its CRDs and the agents' templates surviving, nothing else) — and the **own-Flux scenario**: Flux's source-controller and helm-controller from the upstream manifest, the chart through a `HelmRelease` with `components.flux.enabled: false` (no operator, no second helm-controller, the Flux CRDs' field managers untouched, an agent through that Flux on the platform Harness) and the render guard when the value is flipped on. The smoke found the fresh-install deadlock on the kagent namespace fixed in this release.
+**The ATS** (`execute-chart-tests`, a required check; [tests/ats/README.md](tests/ats/README.md)) runs two scenarios on one kind cluster on a 4-vCPU `large` CircleCI machine — a cluster created from [`.ats/kind-config.yaml`](.ats/kind-config.yaml) with the `ClusterTrustBundle`, `ClusterTrustBundleProjection` and `PodCertificateRequest` feature gates Agent Substrate needs: the **smoke** — the quick start on a bare cluster (the Gateway API CRDs, a lab Dex and an in-cluster registry the candidate is pushed to, together with the connectivity chart of the checkout, which the roster is pointed at through `components.agent-platform-connectivity.{repository,versionRange,insecure}`; `helm install --wait` with the bundled engine, muster with its OAuth server, dicebear, connectivity, kagent with its CRD chart, Substrate with its CRD chart, agent-manager, self-management on against that registry), then the adoption, the auth round trip (the 401 discovery chain, a Dex user through the password grant and through muster's login flow), Substrate from the chart under test (its WorkerPool's gVisor worker Ready, the platform Harness), the agent round trips on kagent API v2 (a declarative `AgentTemplate` Ready on the platform Harness; agent-manager's `create_agent` through muster as the Dex user → the chart `1.x` HelmRelease as `kagent-flux` → the `AgentTemplate` Ready on the Harness and the agent's `RemoteMCPServer` Accepted), the drift correction (the platform Harness deleted by hand and back on the kagent release's next reconcile — a requested one, no `forceAt`, no Helm revision — with both templates Ready on it again), the self-management fixpoint and values Secret, the refused `helm upgrade`, and the ordered teardown with the line's keep policy (the `kagent-crds` release uninstalled, its CRDs and the agents' templates surviving, nothing else) — and the **own-Flux scenario**: Flux's source-controller and helm-controller from the upstream manifest, the chart through a `HelmRelease` with `components.flux.enabled: false` (no operator, no second helm-controller, the Flux CRDs' field managers untouched, an agent through that Flux on the platform Harness) and the render guard when the value is flipped on. The smoke found the fresh-install deadlock on the kagent namespace fixed in this release.
 
 Measured (each test logs `TIMING <phase>`; CI = the 4-vCPU `large` machine with cold pulls, CircleCI job 6033 of PR #380; the per-phase table with the Substrate sizing is in [tests/ats/README.md](tests/ats/README.md#the-cluster-kind-with-agent-substrate)): **smoke** — `helm install --wait` 220 s, Substrate ready 0 s after the install returned (the install already waited for kagent behind it), the declarative `AgentTemplate` Ready on the Harness 20 s, agent-manager `create_agent` → HelmRelease Ready → `AgentTemplate` Ready + `RemoteMCPServer` Accepted 16 s, the fixpoint wait fills the rest of two 1-minute intervals after the adoption, `helm upgrade` refused in 1 s, `helm uninstall --wait` 31 s (five waves; the kagent namespace is kept); the smoke's pytest run 6:58. **functional** — the chart through the cluster's Flux 289 s, the agent through that Flux 36 s, the guard firing 51 s after the flip, the way back 105 s; the pytest run 8:54. The whole `execute-chart-tests` job: **16:55** (kind create 36 s, the ATS image pull and `uv sync` ~1 min, the two pytest runs the rest). The long pole is the two platform installs — every component image pulled once; Substrate's control plane, the worker image and the Go ADK image the Harness boots are the additions of kagent API v2.