You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
load_model answers not_found while a kserve backend is still installing #312
While a newly registered kserve backend's llm-d controller is still installing, list_presets returns an empty list and load_model answers that the model is "neither cached, served nor a preset". The model is a valid preset; the backend is just not ready yet. The not_found answer sends the caller looking for a wrong model name instead of waiting for the backend.
Proposed solution
When the backend's serving stack (the llm-d controller and its presets) is not ready, list_presets and load_model answer with a backend-not-ready reason that names the backend and what it waits on, instead of an empty list or not_found.
Acceptance criteria
During the controller's installation load_model of a preset model answers a backend-not-ready reason, not not_found.
list_presets says the backend is not ready instead of returning an empty list.
Once the backend is ready the same calls behave as today.
Activity