Closed
Description
Describe the bug
Exporting cuML random forest models to Treelite and inferring using Treelite with no GPUs should work in theory. Related documentation - https://docs.rapids.ai/api/cuml/nightly/pickling_cuml_models.html
It seems to be working for single node GPUs but not for multi-GPUs. It emits all raw probabilities as 0.
I am happy to work on a bug fix with some guidance.
Steps/Code to reproduce bug
Note - Giving example with just one of the samples for illustrative purpose but it emits 0 probabilities for any sample.
Single node GPU - Works as expected.
Training
from cuml.ensemble import RandomForestClassifier as cumlRandomForestClassifier
from sklearn.datasets import load_iris
import numpy as np
import pandas as pd
import cuml
X, y = load_iris(return_X_y=True)
X, y = X.astype(np.float32), y.astype(np.int32)
# Train
clf = cumlRandomForestClassifier(max_depth=8, random_state=0, n_estimators=100)
clf.fit(X, y)
# Persist model to treelite
clf.convert_to_treelite_model().to_treelite_checkpoint('./iris_single_gpu_checkpoint.tl')
Serving with Treelite with no GPUs.
import numpy as np
test_sample = np.array([(6.9, 3.1, 4.9, 1.5)])
import treelite
tl_model_sg = treelite.Model.deserialize('./iris_single_gpu_checkpoint.tl')
output = treelite.gtil.predict(tl_model_sg, test_sample, pred_margin=True)
# Treeline predict output
print(output)
Observed output as expected.
[[0. 0.94 0.06]]
Multi GPU -
Training
from sklearn.datasets import load_iris
import numpy as np
import pandas as pd
from dask.distributed import Client, wait
from dask_cuda import LocalCUDACluster
import cuml, cudf, dask_cudf
import cuml.dask
cluster = LocalCUDACluster(n_workers=1, memory_limit="11GB")
client = Client(cluster)
X, y = load_iris(return_X_y=True)
X, y = X.astype(np.float32), y.astype(np.int32)
X = dask_cudf.from_cudf(cudf.from_pandas(pd.DataFrame(X)), npartitions=1)
y = dask_cudf.from_cudf(cudf.from_pandas(pd.Series(y)), npartitions=1)
X, y = cuml.dask.common.utils.persist_across_workers(client, [X, y])
clf = cuml.dask.ensemble.RandomForestClassifier(max_depth=8,
max_features='auto',
n_bins=16,
n_estimators=100,
client=client,
ignore_empty_partitions=False)
clf.fit(X, y)
# Persist model to treelite
clf.get_combined_model().convert_to_treelite_model().to_treelite_checkpoint('./iris_multi_gpu_checkpoint.tl')
Serving with Treelite with no GPUs.
import numpy as np
test_sample = np.array([(6.9, 3.1, 4.9, 1.5)])
import treelite
tl_model_mg = treelite.Model.deserialize('./iris_multi_gpu_checkpoint.tl')
output = treelite.gtil.predict(tl_model_mg, test_sample, pred_margin=True)
# Treeline predict output
print(output)
Observed output - All zeros except raw probabilities. [Not expected]
[[0. 0. 0.]]
Expected behavior
get_combined_model() method should be merging dask-worker local trees to generate a combined model. After this point the exported treelite model should behave same as single GPU trained model. Error doesn't seem to be coming from Treelite because it is working for single GPU model. Overall, if cuML can combine multi GPU trees correctly then treelite inference should work in similar way.
Please correct if my understanding of MNMG RF training and Treelite serving is incorrect.
Environment details (please complete the following information):
- Environment location: Docker
- Linux Distro/Architecture: Ubuntu 20.04.6 LTS
- GPU Model/Driver: A100 Tensor core 40GB, Driver version=515.48.07
- CUDA: 11.7
- Method of cuDF & cuML install: conda
# packages in environment at /opt/conda/envs/rapids-23.02:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
aiobotocore 2.5.0 pypi_0 pypi
aiohttp 3.8.4 pypi_0 pypi
aioitertools 0.11.0 pypi_0 pypi
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
anyio 3.6.2 pyhd8ed1ab_0 conda-forge
aom 3.5.0 h27087fc_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 pypi_0 pypi
arrow-cpp 10.0.1 ha770c72_17_cpu conda-forge
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.6.26 hf365957_1 conda-forge
aws-c-cal 0.5.21 h48707d8_2 conda-forge
aws-c-common 0.8.14 h0b41bf4_0 conda-forge
aws-c-compression 0.2.16 h03acc5a_5 conda-forge
aws-c-event-stream 0.2.20 h00877a2_4 conda-forge
aws-c-http 0.7.6 hf342b9f_0 conda-forge
aws-c-io 0.13.19 h5b20300_3 conda-forge
aws-c-mqtt 0.8.6 hc4349f7_12 conda-forge
aws-c-s3 0.2.7 h909e904_1 conda-forge
aws-c-sdkutils 0.1.8 h03acc5a_0 conda-forge
aws-checksums 0.1.14 h03acc5a_5 conda-forge
aws-crt-cpp 0.19.8 hf7fbfca_12 conda-forge
aws-sdk-cpp 1.10.57 h17c43bd_8 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.12.2 pyha770c72_0 conda-forge
bleach 6.0.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.3 hafa529b_0 conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
boost-cpp 1.78.0 h5adbc97_2 conda-forge
boto3 1.24.33 pypi_0 pypi
botocore 1.29.76 pypi_0 pypi
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
brotlipy 0.7.0 pypi_0 pypi
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
c-blosc2 2.8.0 hf91038e_1 conda-forge
ca-certificates 2022.12.7 ha878542_0 conda-forge
cachetools 5.3.0 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 ha61ee94_1014 conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 pypi_0 pypi
cfitsio 4.2.0 hd9d235c_0 conda-forge
charls 2.3.4 h9c3ff4c_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 pypi_0 pypi
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
colorcet 3.0.1 pyhd8ed1ab_0 conda-forge
confluent-kafka 1.7.0 pypi_0 pypi
contourpy 1.0.7 pypi_0 pypi
cryptography 40.0.1 pypi_0 pypi
cubinlinker 0.2.2 pypi_0 pypi
cucim 23.2.0 pypi_0 pypi
cuda-profiler-api 11.8.86 0 conda-nvidia
cuda-python 11.8.1 pypi_0 pypi
cudatoolkit 11.8.0 h37601d7_11 conda-forge
cudf 23.2.0 pypi_0 pypi
cudf-kafka 23.2.0 pypi_0 pypi
cudf_kafka 23.02.00 py310_g5ad4a85b9d_0 conda-rapidsai
cugraph 23.2.0+0.g450c25b8.dirty pypi_0 pypi
cuml 23.2.0 pypi_0 pypi
cupy 11.6.0 pypi_0 pypi
curl 7.88.1 hdc1c0ab_1 conda-forge
cusignal 23.02.00 py310_g659a6fa_0 conda-rapidsai
cuspatial 23.2.0 pypi_0 pypi
custreamz 23.2.0 pypi_0 pypi
cuxfilter 23.2.0 pypi_0 pypi
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cyrus-sasl 2.1.27 h9033bb2_6 conda-forge
cytoolz 0.12.0 pypi_0 pypi
dask 2023.1.1 pyhd8ed1ab_0 conda-forge
dask-core 2023.1.1 pyhd8ed1ab_0 conda-forge
dask-cuda 23.2.1 pypi_0 pypi
dask-cudf 23.2.0 pypi_0 pypi
dask-glm 0.2.0 pypi_0 pypi
dask-ml 2023.3.24 pypi_0 pypi
datashader 0.13.1a py_0 conda-rapidsai
datashape 0.5.4 py_1 conda-forge
dav1d 1.0.0 h166bdaf_1 conda-forge
debugpy 1.6.7 pypi_0 pypi
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distributed 2023.1.1 pyhd8ed1ab_0 conda-forge
dlpack 0.5 h9c3ff4c_0 conda-forge
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
expat 2.5.0 hcb278e6_1 conda-forge
faiss-proc 1.0.0 cuda conda-rapidsai
fastavro 1.7.3 pypi_0 pypi
fastrlock 0.8 pypi_0 pypi
fiona 1.8.22 pypi_0 pypi
flit-core 3.8.0 pyhd8ed1ab_0 conda-forge
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.3 pypi_0 pypi
freetype 2.12.1 hca18f0e_1 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
frozenlist 1.3.3 pypi_0 pypi
fsspec 2023.4.0 pyh1a96a4e_0 conda-forge
gdal 3.5.3 pypi_0 pypi
geopandas 0.12.2 pyhd8ed1ab_0 conda-forge
geopandas-base 0.12.2 pyha770c72_0 conda-forge
geos 3.11.1 h27087fc_0 conda-forge
geotiff 1.7.1 h7157cca_5 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
hdf4 4.2.15 h9772cbc_5 conda-forge
hdf5 1.12.2 nompi_h4df4325_101 conda-forge
heapdict 1.0.1 py_0 conda-forge
holoviews 1.15.3 pyhd8ed1ab_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagecodecs 2022.12.24 pypi_0 pypi
imageio 2.27.0 pyh24c5eb1_0 conda-forge
importlib-metadata 6.3.0 pyha770c72_0 conda-forge
importlib_metadata 6.3.0 hd8ed1ab_0 conda-forge
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
ipykernel 6.14.0 pypi_0 pypi
ipython 8.4.0 pypi_0 pypi
ipywidgets 8.0.6 pyhd8ed1ab_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jmespath 1.0.1 pypi_0 pypi
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
json-c 0.16 hc379101_0 conda-forge
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
jupyter-core 5.3.0 pypi_0 pypi
jupyter-server-proxy 3.2.2 pyhd8ed1ab_0 conda-forge
jupyter_client 8.1.0 pyhd8ed1ab_0 conda-forge
jupyter_core 5.3.0 py310hff52083_0 conda-forge
jupyter_events 0.6.3 pyhd8ed1ab_0 conda-forge
jupyter_server 2.5.0 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 3.0.7 pyhd8ed1ab_0 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
kealib 1.5.0 ha7026e8_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 pypi_0 pypi
krb5 1.20.1 h81ceb04_0 conda-forge
lcms2 2.15 hfd0df8a_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20230125.0 cxx17_hcb278e6_1 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libarrow 10.0.1 h51ec05e_17_cpu conda-forge
libavif 0.11.1 h5cdd6b5_0 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcublas 11.11.3.6 0 conda-nvidia
libcublas-dev 11.11.3.6 0 conda-nvidia
libcucim 23.02.00 cuda11_gb8cfaa2_0 conda-rapidsai
libcudf 23.02.00 cuda11_g5ad4a85b9d_0 conda-rapidsai
libcudf_kafka 23.02.00 g5ad4a85b9d_0 conda-rapidsai
libcufft 10.9.0.58 0 conda-nvidia
libcugraph 23.02.00 cuda11_g450c25b8_0 conda-rapidsai
libcugraph_etl 23.02.00 cuda11_g450c25b8_0 conda-rapidsai
libcugraphops 23.02.01 cuda11_g505ea813_0 conda-nvidia
libcuml 23.02.00 cuda11_g3356f05bd_0 conda-rapidsai
libcumlprims 23.02.00 cuda11_g0c4a914_0 conda-nvidia
libcurand 10.3.0.86 0 conda-nvidia
libcurand-dev 10.3.0.86 0 conda-nvidia
libcurl 7.88.1 hdc1c0ab_1 conda-forge
libcusolver 11.4.1.48 0 conda-nvidia
libcusolver-dev 11.4.1.48 0 conda-nvidia
libcusparse 11.7.5.86 0 conda-nvidia
libcusparse-dev 11.7.5.86 0 conda-nvidia
libcuspatial 23.02.00 cuda11_g6fe3841_0 conda-rapidsai
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.14 h166bdaf_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 h28343ad_4 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libfaiss 1.7.2 cuda118h2d43ea4_4_cuda conda-rapidsai
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgcrypt 1.10.1 h166bdaf_0 conda-forge
libgdal 3.5.3 h05f8703_11 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.74.1 h606061b_1 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libgoogle-cloud 2.8.0 h0bc5f78_1 conda-forge
libgpg-error 1.46 h620e276_0 conda-forge
libgrpc 1.52.1 hcf146ea_1 conda-forge
libgsasl 1.8.0 2 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libkml 1.3.0 h37653c0_1015 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm11 11.1.0 he0ac6c6_5 conda-forge
libnetcdf 4.8.1 nompi_h261ec11_106 conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libntlm 1.4 h7f98852_1002 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.2 hb675445_0 conda-forge
libprotobuf 3.21.12 h3eb15da_0 conda-forge
libraft-distance 23.02.00 cuda11_g69dce2d4_0 conda-rapidsai
libraft-headers 23.02.00 cuda11_g69dce2d4_0 conda-rapidsai
libraft-nn 23.02.00 cuda11_g69dce2d4_0 conda-rapidsai
librdkafka 1.7.0 hb1989a6_1 conda-forge
librmm 23.02.00 cuda11_g48e8f2a8_0 conda-rapidsai
librttopo 1.1.0 ha49c73b_12 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialindex 1.9.3 h9c3ff4c_4 conda-forge
libspatialite 5.0.1 h7c8129e_22 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libthrift 0.18.1 h5e4af38_0 conda-forge
libtiff 4.5.0 h82bc61c_0 conda-forge
libutf8proc 2.8.0 h166bdaf_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libuv 1.44.2 h166bdaf_0 conda-forge
libwebp 1.2.4 h1daa5a0_1 conda-forge
libwebp-base 1.2.4 h166bdaf_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxgboost 1.7.1dev.rapidsai23.02 cuda_11_2 conda-rapidsai
libxml2 2.10.3 hca2bb57_4 conda-forge
libzip 1.9.2 hc929e4a_1 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
llvmlite 0.39.1 pypi_0 pypi
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.2 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge
markdown 3.4.3 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.2 pypi_0 pypi
matplotlib 3.7.1 pypi_0 pypi
matplotlib-base 3.7.1 py310he60537e_0 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
mistune 2.0.5 pyhd8ed1ab_0 conda-forge
msgpack 1.0.5 pypi_0 pypi
msgpack-python 1.0.5 py310hdf3cbec_0 conda-forge
multidict 6.0.4 pypi_0 pypi
multipledispatch 0.6.0 py_0 conda-forge
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
nbclient 0.7.3 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.3.1 pyhd8ed1ab_0 conda-forge
nbformat 5.8.0 pyhd8ed1ab_0 conda-forge
nccl 2.15.1.1 h0800d71_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
nodejs 18.15.0 h8d033a5_0 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.89 he45b914_0 conda-forge
numba 0.56.4 pypi_0 pypi
numpy 1.23.5 pypi_0 pypi
nvtx 0.2.3 pypi_0 pypi
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.1.0 h0b41bf4_0 conda-forge
orc 1.8.3 hfdbbad2_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 pypi_0 pypi
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
panel 0.14.1 pyhd8ed1ab_0 conda-forge
param 1.13.0 pyh1a96a4e_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.4.0 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pexpect 4.8.0 pyh1a96a4e_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.4.0 pypi_0 pypi
pip 23.0.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
platformdirs 3.2.0 pyhd8ed1ab_0 conda-forge
pooch 1.7.0 pyha770c72_3 conda-forge
poppler 22.12.0 h091648b_1 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 15.2 h3248436_0 conda-forge
proj 9.1.0 h8ffa02c_1 conda-forge
prometheus_client 0.16.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.38 pyha770c72_0 conda-forge
protobuf 4.21.12 pypi_0 pypi
psutil 5.9.4 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptxcompiler 0.7.0 pypi_0 pypi
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
py-xgboost 1.7.1dev.rapidsai23.02 cuda_11_py310_2 conda-rapidsai
pyarrow 10.0.1 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pydeck 0.5.0 pyh9f0ad1d_0 conda-forge
pyee 8.1.0 pyhd8ed1ab_0 conda-forge
pygments 2.15.0 pyhd8ed1ab_0 conda-forge
pylibcugraph 23.2.0+0.g450c25b8.dirty pypi_0 pypi
pylibraft 23.2.0 pypi_0 pypi
pynvml 11.4.1 pyhd8ed1ab_0 conda-forge
pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyppeteer 1.0.2 pyhd8ed1ab_0 conda-forge
pyproj 3.4.0 pypi_0 pypi
pyrsistent 0.19.3 pypi_0 pypi
pysocks 1.7.1 pypi_0 pypi
python 3.10.10 he550d4f_0_cpython conda-forge
python-confluent-kafka 1.7.0 py310h6acc77f_2 conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.16.3 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
pyviz_comms 2.2.1 pyhd8ed1ab_1 conda-forge
pywavelets 1.4.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
pyzmq 25.0.2 pypi_0 pypi
raft-dask 23.2.0 pypi_0 pypi
rapids 23.02.00 cuda11_py310_g87dd3f4_144 conda-rapidsai
rapids-xgboost 23.02.00 cuda11_py310_g87dd3f4_144 conda-rapidsai
re2 2023.02.02 hcb278e6_0 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.28.2 pyhd8ed1ab_1 conda-forge
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rmm 23.2.0 pypi_0 pypi
rtree 1.0.1 pypi_0 pypi
s2n 1.3.41 h3358134_0 conda-forge
s3fs 2023.4.0 pypi_0 pypi
s3transfer 0.6.0 pypi_0 pypi
scikit-image 0.19.3 pypi_0 pypi
scikit-learn 1.2.2 pypi_0 pypi
scipy 1.10.1 pypi_0 pypi
send2trash 1.8.0 pyhd8ed1ab_0 conda-forge
setuptools 67.6.1 pyhd8ed1ab_0 conda-forge
shapely 2.0.1 pypi_0 pypi
simpervisor 0.4 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
spdlog 1.8.5 h4bd325d_1 conda-forge
sqlite 3.40.0 h4ff8645_0 conda-forge
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
streamz 0.6.4 pyh6c4a22f_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
terminado 0.15.0 pypi_0 pypi
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge
tiledb 2.13.2 hd532e3d_0 conda-forge
tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.2 pypi_0 pypi
tqdm 4.65.0 pyhd8ed1ab_1 conda-forge
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
treelite 3.1.0 pypi_0 pypi
treelite-runtime 3.1.0 pypi_0 pypi
typing-extensions 4.5.0 hd8ed1ab_0 conda-forge
typing_extensions 4.5.0 pyha770c72_0 conda-forge
tzcode 2023c h0b41bf4_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
ucx 1.13.1 h538f049_1 conda-forge
ucx-proc 1.0.0 gpu conda-rapidsai
ucx-py 0.30.0 pypi_0 pypi
unicodedata2 15.0.0 pypi_0 pypi
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 1.5.1 pyhd8ed1ab_0 conda-forge
websockets 10.4 pypi_0 pypi
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
widgetsnbextension 4.0.7 pyhd8ed1ab_0 conda-forge
wrapt 1.15.0 pypi_0 pypi
xarray 2023.3.0 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.4 h55805fa_1 conda-forge
xgboost 1.7.1 pypi_0 pypi
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.8.4 h0b41bf4_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xyzservices 2023.2.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.8.2 pypi_0 pypi
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zfp 1.0.0 h27087fc_3 conda-forge
zict 2.2.0 pyhd8ed1ab_0 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zlib-ng 2.0.7 h0b41bf4_0 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
Additional context
Add any other context about the problem here.