Skip to content

Commit

Permalink
update cudf to 22.04, cuda to 11.5 (nv-morpheus#148)
Browse files Browse the repository at this point in the history
Contributes to nv-morpheus#144 (22.06 build args).
Closes nv-morpheus#89 (update cudf).

This PR updates cudf to 22.04 and cuda to 11.5, which is required to compile against cudf 22.04 due to decimal128 support.

Some documentation may still need updating.

Authors:
  - Christopher Harris (https://github.com/cwharris)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: nv-morpheus#148
  • Loading branch information
cwharris authored Jun 13, 2022
1 parent 33a938c commit 3354cf0
Show file tree
Hide file tree
Showing 26 changed files with 284 additions and 707 deletions.
31 changes: 13 additions & 18 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,17 @@ The following instructions are for developers who are getting started with the M

All of the following instructions assume several variables have been set:
- `MORPHEUS_ROOT`: The Morpheus repository has been checked out at a location specified by this variable. Any non-absolute paths are relative to `MORPHEUS_ROOT`.
- `PYTHON_VER`: The desired Python version. Minimum required is 3.8
- `RAPIDS_VER`: The desired RAPIDS version for all RAPIDS libraries including cuDF and RMM. This is also used for Triton. If in doubt use `21.10`
- `CUDA_VER`: The desired CUDA version to use. If in doubt use `11.4`
- `PYTHON_VER`: The desired Python version. Minimum required is `3.8`
- `RAPIDS_VER`: The desired RAPIDS version for all RAPIDS libraries including cuDF and RMM. This is also used for Triton. If in doubt use `22.04`
- `CUDA_VER`: The desired CUDA version to use. If in doubt use `11.5`


### Clone the repository and pull large file data from Git LFS

```bash
export PYTHON_VER=3.8
export RAPIDS_VER=21.10
export CUDA_VER=11.4
export RAPIDS_VER=22.04
export CUDA_VER=11.5
export MORPHEUS_ROOT=$(pwd)/morpheus
git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
cd $MORPHEUS_ROOT
Expand Down Expand Up @@ -135,18 +135,13 @@ This workflow utilizes a docker container to set up most dependencies ensuring a
python 3.8.13 py3.8.13_dbg_morpheus local
```
1. Note: This does not build any Morpheus or Neo code and defers building the code until the entire repo can be mounted into a running container. This allows for faster incremental builds during development.
2. Set up `ssh-agent` to allow container to pull from private repos
```bash
eval `ssh-agent -s`
ssh-add
```
3. Run the development container
2. Run the development container
```bash
./docker/run_container_dev.sh
```
1. The container tag follows the same rules as `build_container_dev.sh` and will default to the current `YYMMDD`. Specify the desired tag with `DOCKER_IMAGE_TAG`. i.e. `DOCKER_IMAGE_TAG=my_tag ./docker/run_container_dev.sh`
2. This will automatically mount the current working directory to `/workspace`.
3. Some of the validation tests require launching a triton docker container within the morpheus container. To enable this you will need to grant the morpheus contrainer access to your host OS's docker socket file with:
3. Some of the validation tests require launching a triton docker container within the morpheus container. To enable this you will need to grant the morpheus container access to your host OS's docker socket file with:
```bash
DOCKER_EXTRA_ARGS="-v /var/run/docker.sock:/var/run/docker.sock" ./docker/run_container_dev.sh
```
Expand All @@ -155,17 +150,17 @@ This workflow utilizes a docker container to set up most dependencies ensuring a
./docker/install_docker.sh
```
4. Compile Morpheus
3. Compile Morpheus
```bash
./scripts/compile.sh
```
This script will run both CMake Configure with default options and CMake build.
5. Install Morpheus
4. Install Morpheus
```bash
pip install -e /workspace
```
Once Morpheus has been built, it can be installed into the current virtual environment.
6. [Run Morpheus](./README.md#running-morpheus)
5. [Run Morpheus](./README.md#running-morpheus)
```bash
morpheus run pipeline-nlp ...
```
Expand Down Expand Up @@ -196,8 +191,8 @@ Note: These instructions assume the user is using `mamba` instead of `conda` sin
1. Setup env variables and clone the repo:
```bash
export PYTHON_VER=3.8
export RAPIDS_VER=21.10
export CUDA_VER=11.4
export RAPIDS_VER=22.04
export CUDA_VER=11.5
export MORPHEUS_ROOT=$(pwd)/morpheus
git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
cd $MORPHEUS_ROOT
Expand Down Expand Up @@ -321,4 +316,4 @@ Ex:
## Attribution
Portions adopted from https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md \
Portions adopted from https://github.com/dask/dask/blob/master/docs/source/develop.rst
Portions adopted from https://github.com/dask/dask/blob/master/docs/source/develop.rst
12 changes: 6 additions & 6 deletions ci/Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ pipeline {
}
environment {
PYTHON_VER = '3.8'
RAPIDS_VER = '21.10'
CUDA_VER = '11.4'
RAPIDS_VER = '22.04'
CUDA_VER = '11.5'
HOME = "${WORKSPACE}"
GH_TOKEN = credentials('gputester-github-token')
BUILD_TYPE = sh(returnStdout: true, script: 'rapids-build-type')
Expand All @@ -24,7 +24,7 @@ pipeline {
when { environment name: 'BUILD_TYPE', value: 'pull-request' }
agent {
docker {
image 'gpuci/rapidsai-driver:21.10-cuda11.4-devel-ubuntu20.04-py3.8'
image 'gpuci/rapidsai-driver:22.04-cuda11.5-devel-centos7-py3.8'
label 'cpu'
}
}
Expand All @@ -48,7 +48,7 @@ pipeline {
}
agent {
docker {
image 'gpuci/rapidsai:21.10-cuda11.4-devel-ubuntu20.04-py3.8'
image 'gpuci/rapidsai:22.04-cuda11.5-devel-centos7-py3.8'
label 'driver-495'
args '--runtime "nvidia" -e "NVIDIA_VISIBLE_DEVICES=$EXECUTOR_NUMBER"'
}
Expand Down Expand Up @@ -81,7 +81,7 @@ pipeline {
}
agent {
docker {
image 'gpuci/rapidsai:21.10-cuda11.4-devel-ubuntu20.04-py3.8'
image 'gpuci/rapidsai:22.04-cuda11.5-devel-centos7-py3.8'
label 'driver-495'
args '--runtime "nvidia" -e "NVIDIA_VISIBLE_DEVICES=$EXECUTOR_NUMBER"'
}
Expand Down Expand Up @@ -112,7 +112,7 @@ pipeline {
}
agent {
docker {
image 'gpuci/rapidsai:21.10-cuda11.4-devel-ubuntu20.04-py3.8'
image 'gpuci/rapidsai:22.04-cuda11.5-devel-centos7-py3.8'
label 'driver-495'
args '--runtime "nvidia" -e "NVIDIA_VISIBLE_DEVICES=$EXECUTOR_NUMBER"'
}
Expand Down
53 changes: 0 additions & 53 deletions ci/conda/recipes/cudf/001-external-table.patch

This file was deleted.

50 changes: 0 additions & 50 deletions ci/conda/recipes/cudf/build.sh

This file was deleted.

24 changes: 0 additions & 24 deletions ci/conda/recipes/cudf/conda_build_config.yaml

This file was deleted.

96 changes: 0 additions & 96 deletions ci/conda/recipes/cudf/meta.yaml

This file was deleted.

Loading

0 comments on commit 3354cf0

Please sign in to comment.