mlserver >= 1.3 doesn't work with MLflow

[harupy]

After the release of mlserver==1.3.1, one of our tests in https://github.com/mlflow/mlflow started failing with the following error. We're investigating the cause.

https://github.com/mlflow/mlflow/actions/runs/4716614587/jobs/8364498235?pr=8241#step:7:8138

2023-04-17 03:00:01,885 [mlserver.parallel] DEBUG - Starting response processing loop...
2023-04-17 03:00:01,894 [mlserver.rest] INFO - HTTP server running on http://0.0.0.0:8080/
2023-04-17 03:00:01,943 [mlserver.metrics] INFO - Metrics server running on http://0.0.0.0:8082/
2023-04-17 03:00:01,944 [mlserver.metrics] INFO - Prometheus scraping endpoint can be accessed on http://0.0.0.0:8082/metrics
2023-04-17 03:00:01,963 [mlserver.grpc] INFO - gRPC server running on http://0.0.0.0:8081/
INFO:     172.17.0.1:59636 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59644 - "GET /ping HTTP/1.1" 404 Not Found
2023-04-17 03:00:03,801 [mlserver] INFO - Loaded model 'mlflow-model' succesfully.
2023-04-17 03:00:03,802 [mlserver] INFO - Loaded model 'mlflow-model' succesfully.
2023-04-17 03:00:03,804 [mlserver] INFO - Loaded model 'mlflow-model' succesfully.
INFO:     172.17.0.1:59646 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59660 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59674 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59690 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59702 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59708 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59712 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:59718 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42498 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42512 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42528 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42530 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42536 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42544 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42546 - "GET /ping HTTP/1.1" 404 Not Found
INFO:     172.17.0.1:42552 - "GET /ping HTTP/1.1" 404 Not Found
...

The error above indicates that the /ping endpoint is not registered.

[harupy]

#1044 might be related

[adriangonz]

Hey @harupy ,

Massive thanks for raising this one!

We'll have a look once we get a chance. It should be possible to replicate just by running the tests/models/test_cli.py::test_build_docker_with_env_override test within the MLflow repo, right?

[harupy]

@adriangonz Thanks for the reply! Yes, pytest 'tests/models/test_cli.py::test_build_docker[True]' or pytest 'tests/models/test_cli.py::test_build_docker_with_env_override[True]' or pytest 'tests/test_cli.py::test_mlflow_models_serve[True]' should be able to reproduce the error.

[harupy]

@adriangonz

Dockerfile that can reproduce the issue:

FROM python:3.8

WORKDIR /mlflow
RUN git clone --branch master --depth 1 https://github.com/mlflow/mlflow.git /mlflow
RUN pip install -e .
RUN pip install -r requirements/test-requirements.txt
RUN pip install mlserver==1.3.1 mlserver-mlflow==1.3.1
RUN pip list | grep mlserver
RUN pytest 'tests/test_cli.py::test_mlflow_models_serve[True]' -s

[adriangonz]

Hey @harupy ,

Just as heads up, you were 100% right - there was an issue introduced in #1044 . Good spot!

We've just pushed a patch for review (#1104), which should do the trick for most cases. There's still an edge case with dynamic custom envs - discussed in #1103 - which should get patched soon. However, that will only get triggered when a new flag in the settings is used (which shouldn't be the case for MLflow models started from the mlflow CLI or run in SCv1 / KServe).

[harupy]

@adriangonz Thanks for the fix! Any plan to release 1.3.2?

[adriangonz]

Sure thing @harupy - I'm checking out a couple other remnants from the initial 1.3 release and will trigger a 1.3.2 once that's done.