Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Separate Pipeline type inference/checking & MRC pipeline construction #1233

Merged
merged 187 commits into from
Oct 19, 2023
Merged

Separate Pipeline type inference/checking & MRC pipeline construction #1233

merged 187 commits into from
Oct 19, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Oct 2, 2023 •
edited
Loading

Description

  • Builds upon changes in PR Draft: Separate Pipeline type inference/checking & MRC pipeline construction #1175
  • Type inference/checking moved to Pipeline.pre_build and doesn't require MRC
  • Wraps input/output types between stages into a StageSchema class which can later be expanded to describe tensors and dataframe columns.
  • StreamWrapper renamed to BaseStage.
  • BaseStage defines a new abstract method compute_schema
  • Fix unrelated bug in generate_viz_frames_stage.py where the overwrite argument was ignored leading to a failed assert.
  • Calling add_edge, add_segment_edge, or add_stage after calling build will trigger a failed assert.

fixes #229
fixes #230

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

dagardner-nv and others added 30 commits August 1, 2023 16:29
@dagardner-nv
wip
0d1aeb9
@dagardner-nv
Don't hold a reference to the pipeline outside of the build method, e…
2a6e65e 
…xplicitly release reference to the executor after join and stop
@dagardner-nv
Test for issue nv-morpheus#1114
07b2069
@dagardner-nv
Rename log parsing test to not conflict with top-level test_pipe
5b5e3f4
@dagardner-nv
Pylint fixes
7745d98
@dagardner-nv
Don't hold a ref to executor options:
73318c7
@dagardner-nv
Skip test_multi_segment_bad_data_type due to MRC issue nv-morpheus/MR…
364da5a 
…C#360
@dagardner-nv
lint fixes
f85dea5
@dagardner-nv
Merge branch 'branch-23.11' into david-destructors-1114
698b3e4
@dagardner-nv
Ensure that both _storage_dir and _storage_type are defined prior to …
de11367 
…potentially raising an exception, this prevents an attribute error in the destructor
@dagardner-nv
wip
506d95a
@dagardner-nv
Revert "wip"
da1929c 
This reverts commit e949da1.
@dagardner-nv
Add comment explaining
2909695
@dagardner-nv
Call PyGILState_Check first to see if we need to gil, borrowed from M…
7394f23 
…RC's PyObjectWrapper
@dagardner-nv
Add comment explaining the call to PyGILState_Check
0f7500c
@dagardner-nv
Test for MRC 362
34c038f
@dagardner-nv
Insert CR header, and add fixture for disabling garbage collection
8984754
@dagardner-nv
Revert "Add comment explaining the call to PyGILState_Check"
7f2a553 
This reverts commit 0f7500c.
@dagardner-nv
Revert "Call PyGILState_Check first to see if we need to gil, borrowe…
f061115 
…d from MRC's PyObjectWrapper"

This reverts commit 7394f23.
@dagardner-nv
Adopt patched pybind11
33cf054
@dagardner-nv
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
5163bc9 
…avid-destructors-1114
@dagardner-nv
Lint fixes
60d20e3
@dagardner-nv
Merge branch 'branch-23.11' into david-destructors-1114
123d730
@dagardner-nv
Merge branch 'branch-23.11' into david-destructors-1114
5bd3ade
@dagardner-nv
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
07b284a 
…avid-destructors-1114
@dagardner-nv
Merge branch 'branch-23.11' into david-pipe-build-refactor
8e4769f
@dagardner-nv
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
080e0c3 
…avid-pipe-build-refactor
@dagardner-nv
wip
fab2b98
@dagardner-nv
Ditch the mixin, as sources don't have parent types
9a00fbe
@dagardner-nv
Add output_type method to sources
a4cd8b6
@dagardner-nv dagardner-nv changed the title Draft: Separate Pipeline type inference/checking & MRC pipeline construction pt2 Separate Pipeline type inference/checking & MRC pipeline construction Oct 9, 2023
@dagardner-nv dagardner-nv added breaking Breaking change feature request New feature or request improvement Improvement to existing functionality and removed improvement Improvement to existing functionality feature request New feature or request labels Oct 9, 2023
@dagardner-nv dagardner-nv self-assigned this Oct 9, 2023
@dagardner-nv dagardner-nv mentioned this pull request Oct 9, 2023
Closed
2 tasks
mdemoret-nv
mdemoret-nv approved these changes Oct 19, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving based on our many discussions of this PR. Documentation and full testing to come later

@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit a0084e3 into nv-morpheus:branch-23.11 Oct 19, 2023
7 checks passed
@mdemoret-nv mdemoret-nv mentioned this pull request Nov 29, 2023
Merged
rapids-bot bot pushed a commit that referenced this pull request Nov 30, 2023
@mdemoret-nv
Fix the SID Viz workflow shutdown process with the new pipeline shutd…
da3814a 
…own process (#1392)

Due to the changes in #1233, the pipeline shutdown process has been slightly altered. This updates the viz demo to shutdown correctly without error. Also added support for exiting on Ctrl+C

Fixes #1391

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Eli Fajardo (https://github.com/efajardo-nv)
  - David Gardner (https://github.com/dagardner-nv)

URL: #1392
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
breaking Breaking change improvement Improvement to existing functionality
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
2 participants
@dagardner-nv @mdemoret-nv