Update CONTRIBUTING.md & fix file ownership (nv-morpheus#27)

* Ensure the cache directories exist

* Updating build_conda_packages to specify which packages to build

* Add missing nodejs dep

* Create dirs ahead of time to prevent them from being owned by root

* Update instructions

* Set user/group ownership of the .cache dir back to the originating user so it won't be owned by root

* Add instructions for setting up ssh-agent prior to running ./docker/build_conda_packages.sh

* Keep the mkdirs

* Update CONTRIBUTING.md

Co-authored-by: Michael Demoret <42954918+mdemoret-nv@users.noreply.github.com>

* Move chown into the docker build and chown the conda-bld dir

Co-authored-by: Michael Demoret <mdemoret@nvidia.com>
Co-authored-by: Michael Demoret <42954918+mdemoret-nv@users.noreply.github.com>

CONTRIBUTING.md

@@ -55,13 +55,17 @@ The following instructions are for developers who are getting started with the M
 All of the following instructions assume several variables have been set:
  - `MORPHEUS_ROOT`: The Morpheus repository has been checked out at a location specified by this variable. Any non-absolute paths are relative to `MORPHEUS_ROOT`.
  - `PYTHON_VER`: The desired Python version. Minimum required is 3.8
- - `RAPIDS_VER`: The desired RAPIDS version for all RAPIDS libraries including cuDF and RMM. This is also used for Triton.
- - `CUDA_VER`: The desired CUDA version to use.
+ - `RAPIDS_VER`: The desired RAPIDS version for all RAPIDS libraries including cuDF and RMM. This is also used for Triton. If in doubt use `21.10`
+ - `CUDA_VER`: The desired CUDA version to use. If in doubt use `11.4`
+
 
 ### Clone the repository and pull large file data from Git LFS
 
 ```bash
-MORPHEUS_ROOT=$(pwd)/morpheus
+export PYTHON_VER=3.8
+export RAPIDS_VER=21.10
+export CUDA_VER=11.4
+export MORPHEUS_ROOT=$(pwd)/morpheus
 git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
 cd $MORPHEUS_ROOT
 ```
@@ -140,9 +144,9 @@ Note: These instructions assume the user is using `mamba` instead of `conda` sin
 - Pascal architecture or better
 - NVIDIA driver `450.80.02` or higher
 - [CUDA 11.0+](https://developer.nvidia.com/cuda-downloads)
-- `conda` or `mamba`
+- `conda` and `mamba`
   - See the [Getting Started Guide](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) if `conda` is not already installed
-  - [Optional] Install `mamba` to speed up the package solver (highly recommended):
+  - Install `mamba`:
 
       ```bash
       conda activate base
@@ -151,40 +155,62 @@ Note: These instructions assume the user is using `mamba` instead of `conda` sin
 
   - **Note:** `mamba` should only be installed once in the base environment
 
+1. Setup env variables and clone the repo:
+   ```bash
+   export PYTHON_VER=3.8
+   export RAPIDS_VER=21.10
+   export CUDA_VER=11.4
+   export MORPHEUS_ROOT=$(pwd)/morpheus
+   git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
+   cd $MORPHEUS_ROOT
+   ```
 1. Create a new Conda environment
    ```bash
-   export CUDAToolkit_ROOT=/usr/local/cuda-{CUDA_VER}
-   mamba env create -n morpheus -f ./docker/conda/environments/cuda${CUDA_VER}_dev.yml
+   mamba create -n morpheus python=${PYTHON_VER}
    conda activate morpheus
    ```
-   This creates an environment named `morpheus` with all necessary dependencies, and activates that environment.
-2. Build cuDF
+
+   This creates a new environment named `morpheus`, and activates that environment.
+1. Set up `ssh-agent` to allow container to pull from private repos
+   ```bash
+   eval `ssh-agent -s`
+   ssh-add
+   ```
+1. Build and install cuDF conda package
    ```bash
-   # Clone cuDF
-   git clone -b branch-${RAPIDS_VER} --depth 1 https://github.com/rapidsai/cudf ${MORPHEUS_ROOT}/.cache/cudf
-   cd ${MORPHEUS_ROOT}/.cache/cudf
-   # Apply the Morpheus cuDF patch
-   git apply --whitespace=fix ${MORPHEUS_ROOT}/cmake/deps/patches/cudf.patch
-   # Build cuDF libraries
-   ./build.sh --ptds libcudf cudf libcudf_kafka cudf_kafka
-   cd ${MORPHEUS_ROOT}
+   ./docker/build_conda_packages.sh libcudf cudf
+   mamba install -c file:///${MORPHEUS_ROOT}/.conda-bld -c nvidia -c rapidsai -c conda-forge libcudf cudf
    ```
    This will checkout, patch, build and install cuDF with the necessary fixes to allow Morpheus to work smoothly with cuDF DataFrames in C++.
-3. Build Morpheus
+1. Install remaining Morpheus dependencies
+   ```bash
+   mamba env update -n morpheus -f ./docker/conda/environments/cuda${CUDA_VER}_dev.yml
+   ```
+1. Build Morpheus
    ```bash
    ./scripts/compile.sh
    ```
    This script will run both CMake Configure with default options and CMake build.
-4. Install Morpheus
+1. Install Morpheus
    ```bash
    pip install -e ${MORPHEUS_ROOT}
    ```
    Once Morpheus has been built, it can be installed into the current virtual environment.
-5. Install camouflage, needed for the unittests
+1. Test the build (Note: some tests will be skipped)
+   ```bash
+   pytest
+   ```
+1. Optional: Run full end-to-end tests
+   - Our end-to-end tests require the [camouflage](https://testinggospels.github.io/camouflage/) testing framework. Install camouflage with:
+      ```bash
+      npm install -g camouflage-server
+      ```
+
+   Run all tests:
    ```bash
-   npm install -g camouflage-server
+   pytest --run_slow
    ```
-6. Run Morpheus
+1. Run Morpheus
    ```bash
    morpheus run pipeline-nlp ...
    ```

docker/build_conda_packages.sh

@@ -30,7 +30,12 @@ export DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME:-"morpheus"}
 export DOCKER_IMAGE_TAG=${DOCKER_IMAGE_TAG:-"conda_build"}
 export DOCKER_TARGET=${DOCKER_TARGET:-"development"}
 
+CUR_UID=$(id -u ${LOGNAME})
+CUR_GID=$(id -g ${LOGNAME})
+
 MORPHEUS_ROOT=${MORPHEUS_ROOT:-$(git rev-parse --show-toplevel)}
+mkdir -p ${MORPHEUS_ROOT}/.cache/ccache
+mkdir -p ${MORPHEUS_ROOT}/.cache/cpm
 
 echo "Building container"
 # Call the build script to get a container ready to build conda packages
@@ -79,6 +84,7 @@ fi
 BUILD_SCRIPT="${BUILD_SCRIPT}
 export CONDA_ARGS=\"${CONDA_ARGS[@]}\"
 ./ci/conda/recipes/run_conda_build.sh "$@"
+chown -R ${CUR_UID}:${CUR_GID} .cache .conda-bld
 "
 
 echo "Running conda build"

docker/conda/environments/cuda11.4_dev.yml

@@ -52,6 +52,7 @@ dependencies:
     - myst-parser==0.17
     - neo 22.04.*
     - ninja=1.10
+    - nodejs=17.4.0
     - pandas=1.3
     - pip
     - protobuf=3.19