Skip to content

unload_model answers loaded:false while the LLMInferenceService remains #242

Closed

Description

@teemow

Problem

unload_model answered "loaded": false for qwen3-5-4b-hf (backend kserve), but the LLMInferenceService stayed in place and Ready, with its predictor pod running. A delete_node_pool right after it was refused because the pool still served that model:

node pool <pool> still runs 1 busy node(s): node <node> runs model-serving/qwen3-5-4b-hf-kserve-… (1 GPU),
serving 1 model(s): LLMInferenceService model-serving/qwen3-5-4b-hf (Qwen/Qwen3.5-4B) — unload them first

kubectl get llminferenceservice -n model-serving then showed qwen3-5-4b-hf Ready, aged 9m48s: it was the object the preceding load_model had created, so it had never been deleted. On the same pool, an earlier unload_model of the same model removed it as expected, so the failure is intermittent. Observed on an AWS test installation, no model cache configured. The answer carried inventory.reason: "no cache claim is configured; there is no inventory to rescan".

Proposed solution

Have unload_model derive loaded from the serving objects after its delete: the LLMInferenceService gone, or carrying a deletionTimestamp, answered as status: Terminating. A delete that was not issued or that failed must surface as an error, never as loaded: false.

Acceptance criteria

  • unload_model answers loaded: false only when the LLMInferenceService is gone or terminating, and otherwise an error naming the object.
  • A test covers an unload whose delete does not take effect.

Activity

  1. teemow commented on Oct 8, 2026

    @teemow
    MemberAuthor

    Fixed in #277, released as v1.14.1-rc.1. unload_model now reads every LLMInferenceService back after its delete: gone or terminating is an unload; still there without a deletionTimestamp is a conflict naming the object, and the model stays served and wired. Proof: TestStopRefusesWhenTheDeleteDoesNotTakeEffect and TestStopAnswersATerminatingObjectAsStopped, make test green, release artifacts published.

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Done ✅

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions