Skip to content

load_model answers not_found while a kserve backend is still installing #312

Open

Description

@teemow

Problem

While a newly registered kserve backend's llm-d controller is still installing, list_presets returns an empty list and load_model answers that the model is "neither cached, served nor a preset". The model is a valid preset; the backend is just not ready yet. The not_found answer sends the caller looking for a wrong model name instead of waiting for the backend.

Proposed solution

When the backend's serving stack (the llm-d controller and its presets) is not ready, list_presets and load_model answer with a backend-not-ready reason that names the backend and what it waits on, instead of an empty list or not_found.

Acceptance criteria

  • During the controller's installation load_model of a preset model answers a backend-not-ready reason, not not_found.
  • list_presets says the backend is not ready instead of returning an empty list.
  • Once the backend is ready the same calls behave as today.

Written by an agent.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Backlog 📦

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions