fix(connectivity): the Hugging Face cache claim is applied by a post-install,post-upgrade hook Job outside Helm's wait — WaitForFirstConsumer no longer fails the release (giantswarm/agent-platform#483)

CHANGELOG.md

@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **The Hugging Face cache claim no longer fails the connectivity release under a `WaitForFirstConsumer` StorageClass** (giantswarm/agent-platform#483; follows #326; epic giantswarm/giantswarm#37639). With `components.modelServing` on, `modelServing.cache.pvc` (`hf-cache`) rendered as a release resource with no consumer of its own: the first predictor (or download Job) that mounts it is what Binds it, so under kind's `standard` and the fleet's default `gp3` it stayed `Pending`, helm-controller's wait counted it as not ready and the release failed on every install and upgrade (`timeout waiting for: [PersistentVolumeClaim/model-serving/hf-cache status: 'InProgress']`, five attempts, then stalled) — with `kserve-resources` before it and substrate, kagent, agent-manager and model-manager behind it. The claim is now **applied by a `post-install,post-upgrade` hook Job** (`templates/model-serving/cache-pvc.yaml`; the chart's hook include and its identity `<release>-hooks`, whose ClusterRole carries `get`, `create`, `patch` on `persistentvolumeclaims` while the claim is the chart's — never `delete`): `kubectl apply --server-side` under the field manager `agent-platform-connectivity`, the log says `created` or `present` and the phase, and nothing waits for the claim to Bind — it binds where its first consumer schedules, which is what a zonal volume needs (the GPU pool's zone). The applied spec is the one the chart rendered (`hf-cache` in the serving namespace, `helm.sh/resource-policy: keep`, `accessModes`, `size`, `storageClassName` incl. `"-"`, `volumeName`), and so is what a `cache.enabled` flip or an uninstall does — the claim stays, now because it is not Helm's. An installation upgrading from a chart that rendered the claim keeps it (the keep policy) and the hook takes its fields over (`--force-conflicts`); a size change is applied by the next upgrade's hook (the StorageClass has to allow expansion); an immutable field (`storageClassName`, `accessModes`, `volumeName`) fails the hook naming it. Installations without the switch render byte-identically (`make verify-modes`); `make verify-serving-slice` asserts the hook, its claim and identity, the knobs, and that `cache.enabled: false` or an `existingClaim` render none of it; `make verify-wiring` the serving shape without a `PersistentVolumeClaim` object; `make verify-gpu-pool` compares both sides without the claim. Not taken: a consumer pod of the chart's own (it would pin a zonal volume to its own zone and hold an RWO claim), an `Immediate`-binding class (the same zone lottery, and not the fleet's default), a release-wide `disableWait` (blind to the platform's dependency chain).
+
 - **`make verify-gpu-pool` passes on a `GOLDEN_REF` that carries `modelServing.gpuPool`** (#482). The untainted case asserts the serving render is byte-identical to `GOLDEN_REF` (`origin/main` by default) but for the discovery block — with the golden rendered at its own defaults. From the commit that merged #482 onto main that golden is *tainted*, so an untainted head could never equal it and `test-ingress-modes` failed on every branch cut from main, main included. The golden now takes the same `modelServing.gpuPool.taint.key=` override when its `values.yaml` declares the key, and the discovery block is cut from both sides: the assertion is that equal inputs render alike, which is what it was always meant to be. A golden from before #482 still renders at its defaults, unchanged.
 
 ### Added

Makefile.custom.mk

@@ -845,7 +845,7 @@ verify-target: ## Assert one release of this chart per target cluster (giantswar
 	@echo "target cluster shapes verified."
 
 .PHONY: verify-serving-slice
-verify-serving-slice: ## Assert the serving slice (giantswarm/agent-platform#326): examples/serving-slice.yaml renders exactly the five kserve releases and connectivity (no muster, dicebear, valkey, kagent, Backstage, agent-manager, model-manager, agentgateway; the engine off) — kserve-runtime-configs after kserve-llmisvc-crd into the kserve namespace with the well-known configs on, the runtimes off, no registry value and its block held back from connectivity; KServe's ingress-gateway value derived onto kserve-resources (a differing copy fails, an equal one is a no-op, none with the Gateway off); runtimeClassName nvidia forwarded; the target knob adds agentgateway and stamps every kubeConfig. The connectivity chart with the forwarded values: the models Gateway on models.<global.domain> with the wildcard Secret and the external-dns hostname, ONE Strict AgentgatewayPolicy on the Gateway (audience dex-k8s-authenticator, inheritance Override, the issuer), the JWKS backend at the issuer's host on 443 with TLS, the discovery ConfigMap's gateway entry; a Certificate only with tls.issuerRef.name; nothing with modelsGateway.enabled false (the default: the Gateway is the slice's, the profile turns it on); the guards name their key. The two 24 GB presets: schema keys, no image, tools on with a parser, one GPU, <= 24 GiB. The live half (a served LLMInferenceService answers 200 with a person's id_token and 401 without; no bearer in the model server's log) runs on a GPU cluster: README "The serving slice and the models Gateway". HELM selects the binary.
+verify-serving-slice: ## Assert the serving slice (giantswarm/agent-platform#326): examples/serving-slice.yaml renders exactly the five kserve releases and connectivity (no muster, dicebear, valkey, kagent, Backstage, agent-manager, model-manager, agentgateway; the engine off) — kserve-runtime-configs after kserve-llmisvc-crd into the kserve namespace with the well-known configs on, the runtimes off, no registry value and its block held back from connectivity; KServe's ingress-gateway value derived onto kserve-resources (a differing copy fails, an equal one is a no-op, none with the Gateway off); runtimeClassName nvidia forwarded; the target knob adds agentgateway and stamps every kubeConfig. The connectivity chart with the forwarded values: the models Gateway on models.<global.domain> with the wildcard Secret and the external-dns hostname, ONE Strict AgentgatewayPolicy on the Gateway (audience dex-k8s-authenticator, inheritance Override, the issuer), the JWKS backend at the issuer's host on 443 with TLS, the discovery ConfigMap's gateway entry; a Certificate only with tls.issuerRef.name; nothing with modelsGateway.enabled false (the default: the Gateway is the slice's, the profile turns it on); the guards name their key. The two 24 GB presets: schema keys, no image, tools on with a parser, one GPU, <= 24 GiB. The cache claim (#483): no PersistentVolumeClaim object — a post-install,post-upgrade hook Job server-side applies hf-cache (keep, RWO, 500Gi; the class incl. "-", size, volumeName and access-mode knobs) as <release>-hooks with get/create/patch on claims; cache.enabled false or an existing claim render neither hook nor identity. The live half (a served LLMInferenceService answers 200 with a person's id_token and 401 without; no bearer in the model server's log) runs on a GPU cluster: README "The serving slice and the models Gateway". HELM selects the binary.
 	@echo "====> $@ ($(CHART_DIR), $(CONNECTIVITY_DIR))"
 	@python3 tests/verify-serving-slice.py $(CHART_DIR) $(CONNECTIVITY_DIR)
 	@echo "serving slice verified."
@@ -2021,13 +2021,14 @@ verify-wiring: ## Assert the standalone's ported wiring: toggles off = no object
 	@helm template t $(CONNECTIVITY_DIR) $(VM) --set components.modelServing.enabled=true --set modelServing.kserve.requireApi=false >/dev/null 2>&1 || { echo "FAIL: modelServing.kserve.requireApi=false must skip the check"; exit 1; }
 	@helm template t $(CONNECTIVITY_DIR) $(VM) --set components.modelServing.enabled=true --api-versions serving.kserve.io/v1alpha1 --api-versions serving.kserve.io/v1beta1 >/dev/null 2>&1 || { echo "FAIL: a cluster that serves the KServe APIs must satisfy the guard without the components"; exit 1; }
 	@echo "ok: modelServing prerequisite guard"
-	@echo "--> modelServing + kserve components on (fleet shape, kagent on): runtime, namespace, discovery ConfigMap, presets, chat template, cache PVC, the two Kyverno policies, the cilium policies incl. the agent egress, the kserve controller policy"
+	@echo "--> modelServing + kserve components on (fleet shape, kagent on): runtime, namespace, discovery ConfigMap, presets, chat template, the cache claim's hook Job (no PVC object: #483) and its identity, the two Kyverno policies, the cilium policies incl. the agent egress, the kserve controller policy"
 	@helm template t $(CONNECTIVITY_DIR) $(WIRING_SERVING) --set components.kagent.enabled=true --set components.kserve-llmisvc-resources.enabled=true >/tmp/vw-ms.out 2>&1 || { cat /tmp/vw-ms.out; exit 1; }
-	@for pattern in 'kind: ClusterServingRuntime' '^  name: kserve-vllm$$' 'image: "docker.io/vllm/vllm-openai:' '^  name: agent-platform-model-serving$$' 'kind: PersistentVolumeClaim' '^  name: hf-cache$$' 'agent-platform-serving-preset-qwen3-8-27b' 'agent-platform-serving-preset-qwen3-14b' 'name: agent-platform-chat-template-qwen3-8-27b' '^  name: model-serving$$' 'kind: Namespace' 'name: agent-platform-connectivity-model-serving-pods' 'name: agent-platform-connectivity-model-serving-deployments' 'redirectPolicy: true' 'name: agent-platform-connectivity-model-serving-predictor$$' 'name: agent-platform-connectivity-model-serving-download$$' 'name: agent-platform-connectivity-kagent-agents-to-model-serving' 'matchName: huggingface.co' 'matchPattern: "\*"' '- remote-node' 'name: agent-platform-connectivity-kserve-controller' 'name: agent-platform-connectivity-llmisvc-controller' 'control-plane: kserve-controller-manager' 'flavor: cilium' 'preset-source: "shipped"'; do \
+	@for pattern in 'kind: ClusterServingRuntime' '^  name: kserve-vllm$$' 'image: "docker.io/vllm/vllm-openai:' '^  name: agent-platform-model-serving$$' '^  name: t-model-serving-cache$$' 'helm.sh/hook: post-install,post-upgrade' '"kind":"PersistentVolumeClaim"' '"name":"hf-cache","namespace":"model-serving"' 'kubectl apply --server-side' 'resources: \["persistentvolumeclaims"\]' 'agent-platform-serving-preset-qwen3-8-27b' 'agent-platform-serving-preset-qwen3-14b' 'name: agent-platform-chat-template-qwen3-8-27b' '^  name: model-serving$$' 'kind: Namespace' 'name: agent-platform-connectivity-model-serving-pods' 'name: agent-platform-connectivity-model-serving-deployments' 'redirectPolicy: true' 'name: agent-platform-connectivity-model-serving-predictor$$' 'name: agent-platform-connectivity-model-serving-download$$' 'name: agent-platform-connectivity-kagent-agents-to-model-serving' 'matchName: huggingface.co' 'matchPattern: "\*"' '- remote-node' 'name: agent-platform-connectivity-kserve-controller' 'name: agent-platform-connectivity-llmisvc-controller' 'control-plane: kserve-controller-manager' 'flavor: cilium' 'preset-source: "shipped"'; do \
 		grep -q -e "$$pattern" /tmp/vw-ms.out || { echo "FAIL: the model serving render lacks $$pattern"; exit 1; }; \
 	done
 	@[ "$$(grep -c 'agent-platform.giantswarm.io/serving-preset: "true"' /tmp/vw-ms.out)" = "9" ] || { echo "FAIL: expected the 9 shipped presets, got $$(grep -c 'agent-platform.giantswarm.io/serving-preset: "true"' /tmp/vw-ms.out)"; exit 1; }
 	@if grep -q 'kind: NetworkPolicy' /tmp/vw-ms.out; then echo "FAIL: a kubernetes NetworkPolicy rendered under the cilium flavor"; exit 1; fi
+	@if grep -q '^kind: PersistentVolumeClaim' /tmp/vw-ms.out; then echo "FAIL: the cache claim rendered as a release resource (Helm's wait would wait for a Bind only the first predictor brings: #483)"; exit 1; fi
 	@echo "ok: model serving fleet shape"
 	@echo "--> the vanilla shape (no served API groups): kubernetes policies, no Kyverno object, no Cilium object; policies.enabled=true without Kyverno fails"
 	@helm template t $(CONNECTIVITY_DIR) --set 'ingress.parentRefs[0].name=x' --namespace agent-platform --set components.modelServing.enabled=true --set components.kserve-crd.enabled=true --set components.kserve-resources.enabled=true --set components.kagent.enabled=true >/tmp/vw-ms-vanilla.out 2>&1 || { cat /tmp/vw-ms-vanilla.out; exit 1; }
@@ -2046,7 +2047,7 @@ verify-wiring: ## Assert the standalone's ported wiring: toggles off = no object
 	@grep -q 'preset-source: "values"' /tmp/vw-ms-presets.out || { echo "FAIL: the values preset is not labelled as such"; exit 1; }
 	@grep -q 'name: agent-platform-chat-template-qwen3-14b' /tmp/vw-ms-presets.out || { echo "FAIL: the inline chat template ConfigMap is missing"; exit 1; }
 	@grep -q -- '--chat-template=/mnt/chat-template/chat-template.jinja' /tmp/vw-ms-presets.out || { echo "FAIL: the --chat-template flag was not appended"; exit 1; }
-	@if grep -q 'kind: PersistentVolumeClaim' /tmp/vw-ms-presets.out; then echo "FAIL: a PVC rendered next to an existing claim"; exit 1; fi
+	@if grep -q 'PersistentVolumeClaim' /tmp/vw-ms-presets.out; then echo "FAIL: a claim (or its hook) rendered next to an existing claim"; exit 1; fi
 	@grep -q 'claimName: models' /tmp/vw-ms-presets.out || { echo "FAIL: the existing claim is not published"; exit 1; }
 	@helm template t $(CONNECTIVITY_DIR) $(WIRING_SERVING) --set modelServing.shippedPresets.enabled=false 2>/dev/null >/tmp/vw-ms-noship.out; if grep -q 'serving-preset: "true"' /tmp/vw-ms-noship.out; then echo "FAIL: shipped presets rendered while disabled"; exit 1; fi
 	@if helm template t $(CONNECTIVITY_DIR) $(WIRING_SERVING) --set 'modelServing.presets[0].metadata.name=bad' >/dev/null 2>&1; then echo "FAIL: a preset without spec was accepted"; exit 1; fi

README.md

@@ -389,6 +389,8 @@ Model serving on a GPU cluster is its own release of this chart (giantswarm/agen
 
 **The well-known configs.** `components.kserve-runtime-configs` (giantswarm/kserve's `kserve-runtime-configs`, `0.2.x`, `dependsOn: [kserve-llmisvc-crd]`, into the llm-d controller's lookup namespace `kserve`) installs the `LLMInferenceServiceConfig`s an `LLMInferenceService` composes from by `baseRefs` — the mirrored `llm-d-cuda` image among them. The block `kserve-runtime-configs:` turns `kserve.llmisvcConfigs.enabled` on and `kserve.servingruntime.enabled` off (the chart's `ServingRuntime`s are the classic path, which the switch's vLLM `ClusterServingRuntime` already serves); no registry value is passed — the chart's default is gsoci (giantswarm/kserve#78), a private registry sets `kserve-runtime-configs.kserve.llmisvcConfigs.imageRegistry`. The block never travels to the connectivity release (`omitKeys`).
 
+**The cache claim.** `modelServing.cache.pvc` (`hf-cache`, `500Gi`, the cluster's default StorageClass) is applied by a `post-install,post-upgrade` hook of the connectivity release, not rendered as one of its resources (giantswarm/agent-platform#483): the claim has no consumer of its own — the first predictor that mounts it Binds it — and under a `WaitForFirstConsumer` class (kind's `standard`, AWS `gp3`) a release resource that never Binds fails Helm's wait and, with it, every release behind connectivity. Nothing waits for it, it outlives the release (`helm.sh/resource-policy: keep` states the intent), and the default class of the fleet needs no special handling.
+
 **The models Gateway.** `modelServing.modelsGateway` (the slice turns it on; off by default, so a platform release serving on the classic path renders none of it) is one agentgateway `Gateway` in the release namespace, `models` on `models.<cluster>.<base domain>` (`<hostPrefix>.<global.domain>`), TLS from the platform's wildcard Secret (`gatewayApi.gateway.tls.secretName`; or `tls.issuerRef.name` for a cert-manager `Certificate` of the host), the external-dns hostname on its infrastructure. KServe's ingress-gateway value is **derived** onto the `kserve-resources` release (`kserve.controller.gateway.ingressGateway.kserveGateway: <namespace>/models`; a differing copy of the operator's fails the render), so every `LLMInferenceService` route attaches to it, and a served model answers at
 
 ```