Skip to content

modelServing: the consumer-less hf-cache PVC never Binds under WaitForFirstConsumer, so Helm's wait fails the connectivity release whenever the switch is on #483

Description

@teemow

Problem

With components.modelServing on, the connectivity chart renders the Hugging Face cache claim modelServing.cache.pvc (hf-cache, helm.sh/resource-policy: keep) in the serving namespace with no consumer. Under a StorageClass with volumeBindingMode: WaitForFirstConsumer — kind's standard, and the default gp3 of Giant Swarm's AWS clusters — the claim stays Pending until the first predictor mounts it, so helm-controller's wait fails the connectivity release on every install or upgrade with the switch on:

Helm upgrade failed for release agent-platform/agent-platform-connectivity … timeout waiting for: [PersistentVolumeClaim/model-serving/hf-cache status: 'InProgress']

followed by a rollback and a retry (five attempts, then stalled). With the switch on the meta chart makes agent-platform-connectivity depend on kserve-resources, and substrate, kagent, agent-manager and model-manager depend on connectivity — so the whole platform waits behind a claim that cannot Bind. Seen in the agentlab proof of #326 (three timed-out attempts; modelServing.cache.enabled: false as the workaround). The template dates from the standalone's port (cbf3084); the serving slice (#326) is the first shape that installs it under Flux.

Proposed solution

Take the claim out of Helm's wait rather than out of the chart: keep rendering it, but either annotate it so kstatus/helm-controller does not wait for it to Bind (the claim is a data volume, not a readiness signal), create it from a post-install hook without wait, or document that modelServing.cache.pvc.storageClassName must name an Immediate-binding class (least preferred — the default classes of the fleet bind late). Whatever the choice, examples/serving-slice.yaml installs green on a cluster whose default StorageClass binds WaitForFirstConsumer.

Acceptance criteria

  • The connectivity release with components.modelServing.enabled: true reaches Ready on a kind cluster (StorageClass standard, WaitForFirstConsumer) with modelServing.cache.enabled: true, and on an AWS installation with the default gp3.
  • The claim still exists with helm.sh/resource-policy: keep and is Bound by the first predictor that mounts it.
  • make verify-serving-slice (or a sibling check) asserts the render shape chosen.

Related: #326 (the serving slice), bumblebee-plans#46 (D8), epic giantswarm/giantswarm#37639.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    • Status
      Done ✅

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions