Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install test dependencies at the same time as cuml packages. #5781

Merged
merged 4 commits into from
Feb 26, 2024
Merged

Install test dependencies at the same time as cuml packages. #5781

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1770700
Install test dependencies at the same time as cuml packages.
bdice Feb 22, 2024
44dcfc6
Add rapidsai channel to force DAG sort to prioritize local artifact c…
bdice Feb 22, 2024
3a3e1a8
Use rapids-dependency-file-generator with --prepend-channels.
bdice Feb 23, 2024
38ac4a1
Add test_cuml to test_notebooks.
bdice Feb 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ci/release/update-version.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2019-2023, NVIDIA CORPORATION.
# Copyright (c) 2019-2024, NVIDIA CORPORATION.
########################
# cuML Version Updater #
########################
Expand Down Expand Up @@ -65,8 +65,11 @@ sed_runner "s/branch-.*/branch-${NEXT_SHORT_TAG}/g" ci/test_wheel.sh

DEPENDENCIES=(
cudf
cuml
dask-cuda
dask-cudf
libcuml
libcuml-tests
libcumlprims
libraft-headers
libraft
Expand Down
14 changes: 6 additions & 8 deletions ci/test_cpp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,15 @@ set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

rapids-logger "Generate C++ testing dependencies"
rapids-dependency-file-generator \
--output conda \
--file_key test_cpp \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch)" \
--prepend-channels "${CPP_CHANNEL}" | tee env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

Expand All @@ -18,17 +22,11 @@ set +u
conda activate test
set -u

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
rapids-print-env

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}/
mkdir -p "${RAPIDS_TESTS_DIR}"

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
libcuml libcuml-tests

rapids-logger "Check GPU usage"
nvidia-smi

Expand Down
18 changes: 7 additions & 11 deletions ci/test_notebooks.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,19 @@
#!/bin/bash
# Copyright (c) 2020-2023, NVIDIA CORPORATION.
# Copyright (c) 2020-2024, NVIDIA CORPORATION.
set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-logger "Generate Notebook testing dependencies"
rapids-dependency-file-generator \
--output conda \
--file_key test_notebooks \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \
--prepend-channels "${CPP_CHANNEL};${PYTHON_CHANNEL}" | tee env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

Expand All @@ -17,17 +22,8 @@ set +u
conda activate test
set -u

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcuml cuml

rapids-logger "Check GPU usage"
nvidia-smi

Expand Down
20 changes: 8 additions & 12 deletions ci/test_python_common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,20 @@
#!/bin/bash
# Copyright (c) 2022-2023, NVIDIA CORPORATION.
# Copyright (c) 2022-2024, NVIDIA CORPORATION.

set -euo pipefail

. /opt/conda/etc/profile.d/conda.sh

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)

rapids-logger "Generate Python testing dependencies"
rapids-dependency-file-generator \
--output conda \
--file_key test_python \
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" | tee env.yaml
--matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \
--prepend-channels "${CPP_CHANNEL};${PYTHON_CHANNEL}" | tee env.yaml

rapids-mamba-retry env create --force -f env.yaml -n test

Expand All @@ -18,21 +23,12 @@ set +u
conda activate test
set -u

rapids-logger "Downloading artifacts from previous jobs"
CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)
PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python)
rapids-print-env

RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"}
RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${PWD}/coverage-results"}
mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}"

rapids-print-env

rapids-mamba-retry install \
--channel "${CPP_CHANNEL}" \
--channel "${PYTHON_CHANNEL}" \
libcuml cuml

rapids-logger "Check GPU usage"
nvidia-smi

Expand Down
15 changes: 15 additions & 0 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,19 +48,22 @@ files:
output: none
includes:
- cuda_version
- test_libcuml
- test_cpp
test_python:
output: none
includes:
- cuda_version
- py_version
- test_cuml
- test_python
test_notebooks:
output: none
includes:
- cuda_version
- py_run
- py_version
- test_cuml
- test_notebooks
py_build:
output: pyproject
Expand Down Expand Up @@ -376,6 +379,18 @@ dependencies:
- matrix:
packages:
- python>=3.9,<3.11
test_libcuml:
common:
- output_types: conda
packages:
- libcuml==24.4.*
- libcuml-tests==24.4.*
test_cuml:
common:
- output_types: conda
packages:
- libcuml==24.4.*
- cuml==24.4.*
Comment on lines +382 to +393
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would we want this for wheels too?

Copy link
Contributor Author

@bdice bdice Feb 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's not needed. This dependency list is used exclusively for conda CI, to generate the environment with the PR artifacts and test dependencies in one solve. Wheels go through a different installation path and I'm not sure if it's possible to consolidate that environment generation into one step or not. Even if it is possible, it's unlikely that listing cuml-cu12 in the requirements file would be sufficient, since we need to install a specific filename for the wheel that was fetched from CI artifacts.

test_cpp:
common:
- output_types: conda
Expand Down
Loading