Skip to content

kserve: a GPU pool at scale-to-zero has no eligible node, so check_fit is false and load_model refuses — the first model can never be served on it #90

Closed

Description

@teemow

Problem

On gazelle the GPU node pool gpu-l4 (cluster-manager, Karpenter, one g6.xlarge at most) sits at scale-to-zero until a GPU pod is pending — the design of bumblebee-plans#46 (D10: "Karpenter launches it if scaled to zero"). model-manager's kserve backend cannot serve the first model on it:

check_fit {backend: kserve, preset: qwen3-4b-instruct} →
fits: false, reason: "no eligible node: ip-10-0-164-72…: outside the serving node selector (giantswarm.io/machine-pool=gazelle-gpu-l4); …"

fitCheck looks for a node matching the pool selector and the GPU resource; with none, Fits is false and Load / Pull refuse (backend.go: if !plan.Result.Fits { return ErrUnfit }). Nothing ever creates the LLMInferenceService whose pending predictor would make Karpenter launch the node — the pool can only be used once something else has scaled it up (giantswarm/giantswarm#37714).

Proposed solution

Treat an empty GPU pool as a pool that scales from zero: when the discovery ConfigMap's spec.gpuPool.nodeSelector (or the backend document's) is set and no node matches it, the fit check answers fits: true with node: "", budgetSource: pool-scale-from-zero and a reason that says so ("no node in the GPU pool yet (giantswarm.io/machine-pool=gazelle-gpu-l4); the pool scales from zero — the predictor waits for its node, the fit against the accelerator is unverified"), and Load/Pull proceed without a node pin: the predictor (toleration + pool selector from the preset) goes Pending and Karpenter brings the node. The strict answer stays when nodes exist in the pool but none fits, and when there is no pool selector at all ("no accelerator node"). Optional follow-up: cluster-manager publishes the pool's accelerator memory in the discovery ConfigMap so the fit can be verified against it.

Acceptance criteria

  • With a pool selector and zero matching nodes, check_fit returns fits: true naming the scale-from-zero case; load_model creates the LLMInferenceService; list_nodes still shows no node.
  • With matching nodes the behaviour is unchanged (fit against the node's budget).
  • Unit tests for the three cases (no pool selector, empty pool, populated pool).

Plan bumblebee-plans#46 (D5/D10), epic giantswarm/giantswarm#37639.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions