You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
kserve: a GPU pool at scale-to-zero has no eligible node, so check_fit is false and load_model refuses — the first model can never be served on it #90
On gazelle the GPU node pool gpu-l4 (cluster-manager, Karpenter, one g6.xlarge at most) sits at scale-to-zero until a GPU pod is pending — the design of bumblebee-plans#46 (D10: "Karpenter launches it if scaled to zero"). model-manager's kserve backend cannot serve the first model on it:
check_fit {backend: kserve, preset: qwen3-4b-instruct} →
fits: false, reason: "no eligible node: ip-10-0-164-72…: outside the serving node selector (giantswarm.io/machine-pool=gazelle-gpu-l4); …"
fitCheck looks for a node matching the pool selector and the GPU resource; with none, Fits is false and Load / Pull refuse (backend.go: if !plan.Result.Fits { return ErrUnfit }). Nothing ever creates the LLMInferenceService whose pending predictor would make Karpenter launch the node — the pool can only be used once something else has scaled it up (giantswarm/giantswarm#37714).
Proposed solution
Treat an empty GPU pool as a pool that scales from zero: when the discovery ConfigMap's spec.gpuPool.nodeSelector (or the backend document's) is set and no node matches it, the fit check answers fits: true with node: "", budgetSource: pool-scale-from-zero and a reason that says so ("no node in the GPU pool yet (giantswarm.io/machine-pool=gazelle-gpu-l4); the pool scales from zero — the predictor waits for its node, the fit against the accelerator is unverified"), and Load/Pull proceed without a node pin: the predictor (toleration + pool selector from the preset) goes Pending and Karpenter brings the node. The strict answer stays when nodes exist in the pool but none fits, and when there is no pool selector at all ("no accelerator node"). Optional follow-up: cluster-manager publishes the pool's accelerator memory in the discovery ConfigMap so the fit can be verified against it.
Acceptance criteria
With a pool selector and zero matching nodes, check_fit returns fits: true naming the scale-from-zero case; load_model creates the LLMInferenceService; list_nodes still shows no node.
With matching nodes the behaviour is unchanged (fit against the node's budget).
Unit tests for the three cases (no pool selector, empty pool, populated pool).
Plan bumblebee-plans#46 (D5/D10), epic giantswarm/giantswarm#37639.
Activity