[BUG]: Cannot install any cuml version with morpheus

[tanmoyio]

Version

22.12

Which installation method(s) does this occur on?

Docker

Describe the bug.

Here are steps to reproduce the issue

Step 1: Pull the image

sudo docker pull nvcr.io/nvidia/morpheus/morpheus:22.11-runtime

Step 2: Start the container using following command

sudo docker run --rm -ti --runtime=nvidia --gpus=all --net=host nvcr.io/nvidia/morpheus/morpheus:22.11-runtime bash

Step 3: Installing cuml-22.08 using this command

mamba install -c rapidsai -c nvidia -c conda-forge cuml=22.08
it says there are some packages needed to be downgraded. We accepted those changes

Step 4: reverting back to old dask version to make it morpheus aligned

pip install --no-deps dask[distributed]==2022.11.1

Step 5: Trying to importing cudf

More

We have tried similar approach keeping the rapids version alignment with morpheus in our mind, but it fails every time. In the Conda yml for Morpheus and cuml there are some version misalignment we have found. E.g - morpheus uses cuda-python<=1.7.0 but cuml requires cuda-python>=1.7.1.

What are we expecting?

Short term:

It would be great if there are any ways to integrate cuml to existing containers.

Long term:

In long term, we are expecting the entire rapids stack integration inside morpheus. Also there are misalignment of rapids versions with morpheus, (e.g. morpheus-23.01 is still using cudf-22.10). It would be nicer to have latest versions of the rapids stack inside morpheus container.

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[jimux]

I was able to work around this by adding cuml to the conda recipe and rebuilding the release container:

~/morpheus$ git diff
diff --git a/ci/conda/recipes/morpheus/meta.yaml b/ci/conda/recipes/morpheus/meta.yaml
index 4c3679c..37269a0 100644
--- a/ci/conda/recipes/morpheus/meta.yaml
+++ b/ci/conda/recipes/morpheus/meta.yaml
@@ -58,8 +58,11 @@ outputs:
         - cuda-python <=11.7.0
         - cudatoolkit {{ cuda_version }}.*
         - cudf {{ rapids_version }}
+        - cuml {{ rapids_version }}
         - cython >=0.29,<0.30
         - libcudf {{ rapids_version }}
+        - libcusparse <12
+        - libcusolver <=11.4.1.48
         - librdkafka 1.7
         - mrc {{ minor_version }}
         - pip
@@ -74,12 +77,15 @@ outputs:
         - configargparse 1.5.*
         - cuda-python <=11.7.0 # Needed to ensure cudatoolkit and not cuda-toolkit is installed
         - cudf
+        - cuml {{ rapids_version }}
         - cudf_kafka {{ rapids_version }}.*
         - cupy # Version determined from cudf
         - datacompy 0.8.*
         - distributed
         - docker-py 5.0.*
         - grpcio # Version determined from cudf
+        - libcusparse <12
+        - libcusolver <=11.4.1.48
         - libmrc
         - mlflow >1.29,<2
         - mrc

~/morpheus$ docker run --rm -it nvcr.io/nvidia/morpheus/morpheus:v23.01.00-runtime python -c "
import cupy as cp
import cudf
from cuml.cluster import DBSCAN
import morpheus
gdf_float = cudf.DataFrame()
gdf_float['0'] = [1.0, 2.0, 5.0]
gdf_float['1'] = [4.0, 2.0, 1.0]
gdf_float['2'] = [4.0, 2.0, 1.0]
dbscan_float = DBSCAN(eps=1.0, min_samples=1)
dbscan_float.fit(gdf_float)
print(dbscan_float.labels_)
x_gpu = cp.array([1, 2, 3])
print(cp.linalg.norm(x_gpu))"
0    0
1    1
2    2
dtype: int32
3.7416573867739413

Note that the inclusion of libcuspare and libcusolver is due to this issue in RAPIDS: https://docs.rapids.ai/notices/rsn0026/ An upgrade of RAPIDS in Morpheus would fix that.

[tanmoyio]

@dagardner-nv it would be nicer to have 22.12 cuml and 22.12 cudf support

[dagardner-nv]

@tanmoyio unfortunately for the 23.03 release we won't be able to move to 22.12, and will only work with 22.10.

Can you do me a favor and open an issue requesting 22.12?