fix(connectivity): the Hugging Face cache claim is applied by a post-install,post-upgrade hook Job outside Helm's wait — WaitForFirstConsumer no longer fails the release (giantswarm/agent-platform#483)

[teemow]

Closes #483. Follows #326 (the serving slice, whose lab proof surfaced this); epic giantswarm/giantswarm#37639.

Problem

With components.modelServing on, the connectivity chart rendered the Hugging Face cache claim modelServing.cache.pvc (hf-cache, helm.sh/resource-policy: keep) in the serving namespace as a release resource with no consumer of its own. The first predictor (or download Job) that mounts it is what Binds it, so under a StorageClass with volumeBindingMode: WaitForFirstConsumer — kind's standard, the default gp3 of Giant Swarm's AWS clusters — the claim stayed Pending, helm-controller's wait counted it as not ready, and the release failed on every install and upgrade with the switch on:

Helm upgrade failed for release agent-platform/agent-platform-connectivity … timeout waiting for: [PersistentVolumeClaim/model-serving/hf-cache status: 'InProgress']

five attempts, then stalled — with kserve-resources before it and substrate, kagent, agent-manager and model-manager behind it (the #481 lab proof ran with modelServing.cache.enabled: false for that reason).

Solution

The claim is applied by a post-install,post-upgrade hook Job, never rendered as a release resource (templates/model-serving/cache-pvc.yaml; the chart's hook include agent-platform.hooks.job and its identity <release>-hooks, the way the Postgres connection Secrets are derived). Hook objects are outside Helm's wait: the Job server-side-applies the claim (kubectl apply --server-side --force-conflicts --field-manager=agent-platform-connectivity, one JSON line), logs created or present and the phase, and exits — nothing waits for the claim to Bind. It binds where its first consumer schedules, which is what a zonal volume (gp3) needs: the GPU pool's zone, not the zone of whatever pod would have claimed it first.

• The applied spec is the one the chart rendered: hf-cache in the serving namespace, the keep annotation (now documentary — the claim is not Helm's, so an uninstall or a cache.enabled flip leaves it by construction), accessModes, size, storageClassName incl. "-" (the empty class), volumeName; existingClaim renders nothing and publishes the name as before.
• The hook identity's ClusterRole gains get, create, patch on persistentvolumeclaims only while the claim is the chart's — never delete; agent-platform.hooks.enabled counts the cache hook, so the slice (no Substrate, no Postgres) renders the identity for it.
• Upgrades: an installation coming from a chart that rendered the claim keeps it (the keep policy) and the hook takes its fields over; a size change is applied by the next upgrade's hook (the class has to allow expansion); an immutable field (storageClassName, accessModes, volumeName) fails the hook — and the release — naming the field, as before.
• Not taken: a consumer pod of the chart's own (pins a zonal volume to that pod's zone and holds an RWO claim); an Immediate-binding class (the same zone lottery, and not the fleet's default); a release-wide disableWait (blind to the platform's dependency chain).
• Docs: values.yaml (both charts), the connectivity README ("The Hugging Face cache claim"), the root README's serving-slice section, examples/serving-slice.yaml, CHANGELOG; helm-docs regenerated. Installations without the switch render byte-identically.

Verification

make verify-serving-slice
ok: the cache claim: no PersistentVolumeClaim object; a post-install,post-upgrade hook Job server-side applies hf-cache into model-serving (keep, RWO, 500Gi, the default class) as t-hooks, whose ClusterRole carries get/create/patch on claims and never delete
ok: storageClassName, size, volumeName and accessModes reach the applied claim; "-" is the empty class
ok: cache.enabled: false and an existing claim render no claim, no hook and no hook identity; the existing claim is published
make verify-wiring     the model serving render: the hook Job, its identity, the applied claim, and NO `kind: PersistentVolumeClaim` object
make verify-gpu-pool   ok: an empty taint key leaves the serving render (cache claim off on both sides) byte-identical to origin/main but for the discovery block
make verify-modes      ok: default render unchanged
make verify-target     ok: 5 renders byte-identical to origin/main
make verify-all        43 targets green
helm lint              both charts, the serving fixtures: 0 failed

agentlab proof

Second lab (agentlab-483, kind, StorageClass standard — rancher.io/local-path, WaitForFirstConsumer, no expansion), cert-manager v1.19.1 installed as the KServe controllers' prerequisite; the lab's platform release on 4.28.0 first (12 HelmReleases Ready; the lab's --model-manager=false does not switch the default-on component off on 4.28.0, so components.model-manager.enabled: false in the overlay), then platform.chartBranch on this branch's dev build 4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5 with the serving half of examples/serving-slice.yaml and modelServing.cache.enabled: true — the shape #326's proof had to run with the cache off.

• The install with the switch on reaches Ready. The five KServe releases (0.2.4) and agent-platform-connectivity v3 on the dev build: Ready=True UpgradeSucceeded — 17/17 HelmReleases Ready, no crash-looping pod. The hook Job agent-platform-connectivity-model-serving-cache ran once the release's own wait had passed (SuccessfulCreate → Completed within 3 s, its objects removed by hook-succeeded). The claim it applied:

  NAME       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  hf-cache   Pending                                      standard       20s
  Events: WaitForFirstConsumer  waiting for first consumer to be created before binding
  annotations: {"helm.sh/resource-policy":"keep"}   managedFields manager: agent-platform-connectivity
  

  i.e. the claim that failed the release before now sits Pending next to a Ready release.
• The first consumer Binds it. A stand-in for the first predictor (a pod mounting hf-cache at /mnt/models) scheduled, the provisioner ran (ExternalProvisioning → Provisioning → ProvisioningSucceeded), the pod wrote /mnt/models/proof/ok and exited, and the claim reads Bound pvc-a86dc22a… 500Gi RWO standard.
• The upgrade path. A values change (a label on the serving namespace) upgraded connectivity to v4: the hook's log —

  cache claim model-serving/hf-cache present, Bound (it Binds with its first consumer; nothing waits for it)
  

  the release Ready=True UpgradeSucceeded (v4), the claim untouched (Bound, same volume), 17/17 Ready. A reconcile without a values change makes no Helm revision and runs no hook.
• agentlab platform-test: 7 PASS with the slice on (Claude Code → muster → mcp-kubernetes → apiserver; forwarded id_token RBAC; kagent route JWT Strict; agent-manager as the caller; atelet image cache; per-server OAuth; fake fleet off). agentlab down afterwards.

Not exercised in kind: a real predictor on a GPU node binding the claim in the pool's zone (the gp3 half of the acceptance criteria is the same mechanism — the class binds on the first consumer, the release never waits for it).

[circleci-architect]

Published Helm chart agent-platform-connectivity

4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5

Chart	agent-platform-connectivity
Version	4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5
OCI reference	oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity:4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5
Digest	sha256:431107b79f67fd2602aecac2b22e6e5ac256582b6ce760cee640413442e95b9d
Registry	public — gsoci.azurecr.io
Git catalog	giantswarm-test-catalog (index)

Pull this chart

helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform-connectivity --version 4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5

_{Posted by architect-orb · build 8625 · commit 97fe4a5 · updated in place on every push}

[circleci-architect]

Published Helm chart agent-platform

4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5

Chart	agent-platform
Version	4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5
OCI reference	oci://gsoci.azurecr.io/charts/giantswarm/agent-platform:4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5
Digest	sha256:48c4583c382419168722d3f4fdc7ba2d70d6511a1c94b1ec93afa1f5337b6ebb
Registry	public — gsoci.azurecr.io
Git catalog	giantswarm-test-catalog (index)

Pull this chart

helm pull oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version 4.28.1-dev.hf-cache-pvc-wait-483.2026-09-16.09-52-16.h97fe4a5

_{Posted by architect-orb · build 8626 · commit 97fe4a5 · updated in place on every push}