Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jenkins integration #80

Merged
204 commits merged into from
May 23, 2022
Merged

Jenkins integration #80

204 commits merged into from
May 23, 2022

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented May 3, 2022

Updates conda scripts to uses sccache (ccache work-alike with an S3 backend)
Adds 4 Jenkins stages:

  1. check
  2. build
  3. test
  4. docs build

Results are publised into S3 with a 30 day retention police and are available via:
https://downloads.rapids.ai/ci/morpheus/pull-request/<pull-req id>/<git commit>/<arch>/
ex:
https://downloads.rapids.ai/ci/morpheus/pull-request/80/5161d30673f4d5fb34452df5a8ac6f68b8462685/x86_64/

Intermediate Artifacts pushed to S3:

  • conda_env.tar.gz - Morpheus conda environment used by test & documentation stages
  • workspace.tar.bz - Morpheus workspace used by test & documentation stages

Results pushed to S3:

  • docs.tar.bz - Sphinx HTML documentation
  • report_pytest.xml - Pytest junit report
  • report_pytest_coverage.xml - Pytest coverage report
  • conda_build.tar.bz - Morpheus conda build (not currently working)

Conda builds for cudf are cached seperately (even with sscache this step was still took 17 minutes).
S3 cache location at:
https://downloads.rapids.ai/ci/morpheus/cudf/<cuda ver>/<python ver>/<rapids ver>/<latest git commit to ci/conda>/<arch>cudf_conda.tar.bz
ex:
https://downloads.rapids.ai/ci/morpheus/cudf/11.4/3.8/21.10/b7f728e6db4fc6124970935f461a6791818bfb1d/x86_64/cudf_conda.tar.bz

TODO List:

  • Need a test stage
  • Need to run style checks
  • Need to add sccache
  • Need to cache cudf conda packages
  • Need to be able to build without using a gpu
  • Output junit style xml
  • Output docs build
  • Add conda build step
  • Need to figure out a way to cache lfs

Closes #102
Closes #104

@dagardner-nv dagardner-nv added enhancement Additional functionality added to an existing feature 2 - In Progress labels May 3, 2022
@dagardner-nv dagardner-nv added non-breaking Non-breaking change improvement Improvement to existing functionality and removed enhancement Additional functionality added to an existing feature 2 - In Progress labels May 3, 2022
@dagardner-nv dagardner-nv reopened this May 3, 2022
…gpu, I have a branch with the non-inplace build which should fix this
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes looking much better. Approving so we can start running this against PRs.

@mdemoret-nv
Copy link
Contributor

@gpucibot merge

1 similar comment
@mdemoret-nv
Copy link
Contributor

@gpucibot merge

@ghost ghost merged commit c52cb24 into nv-morpheus:branch-22.06 May 23, 2022
@dagardner-nv dagardner-nv deleted the david-ci branch February 12, 2024 23:20
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Broken cudf builds [FEA] Setup CI to use new Jenkins Build
3 participants