Skip to content

Question : Asynchronous capabilities for mlserver mlflow runtime #2219

Open

Description

@ylambruschi

It seems that mlflow runtime only handle predict methods of its model in a synchronous way when called from

model_output = self._model.predict(decoded_payload, params=params)

raw_predictions = self._model.predict(data, params=inference_params)

I know mlflow embedded serve() uses Flask and is not designed to be async, but there is no hard interface contract expecting predict() to be synchronous right ?
Let's say i develop my model and set the predict() method to be async. would it make sense to work on the possibility to await it in mlserver mlflow runtime to make all the chain async ? ( maybe with some option)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions