Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate Pipeline type inference/checking & MRC pipeline construction #1233

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
187 commits
Select commit Hold shift + click to select a range
0d1aeb9
wip
dagardner-nv Aug 1, 2023
2a6e65e
Don't hold a reference to the pipeline outside of the build method, e…
dagardner-nv Aug 1, 2023
07b2069
Test for issue #1114
dagardner-nv Aug 1, 2023
5b5e3f4
Rename log parsing test to not conflict with top-level test_pipe
dagardner-nv Aug 2, 2023
7745d98
Pylint fixes
dagardner-nv Aug 2, 2023
73318c7
Don't hold a ref to executor options:
dagardner-nv Aug 2, 2023
364da5a
Skip test_multi_segment_bad_data_type due to MRC issue https://github…
dagardner-nv Aug 2, 2023
f85dea5
lint fixes
dagardner-nv Aug 2, 2023
698b3e4
Merge branch 'branch-23.11' into david-destructors-1114
dagardner-nv Aug 4, 2023
de11367
Ensure that both _storage_dir and _storage_type are defined prior to…
dagardner-nv Aug 15, 2023
506d95a
wip
dagardner-nv Aug 14, 2023
da1929c
Revert "wip"
dagardner-nv Aug 15, 2023
2909695
Add comment explaining
dagardner-nv Aug 15, 2023
7394f23
Call PyGILState_Check first to see if we need to gil, borrowed from M…
dagardner-nv Aug 15, 2023
0f7500c
Add comment explaining the call to PyGILState_Check
dagardner-nv Aug 15, 2023
34c038f
Test for MRC 362
dagardner-nv Aug 17, 2023
8984754
Insert CR header, and add fixture for disabling garbage collection
dagardner-nv Aug 17, 2023
7f2a553
Revert "Add comment explaining the call to PyGILState_Check"
dagardner-nv Aug 17, 2023
f061115
Revert "Call PyGILState_Check first to see if we need to gil, borrowe…
dagardner-nv Aug 17, 2023
33cf054
Adopt patched pybind11
dagardner-nv Aug 18, 2023
5163bc9
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Aug 18, 2023
60d20e3
Lint fixes
dagardner-nv Aug 18, 2023
123d730
Merge branch 'branch-23.11' into david-destructors-1114
dagardner-nv Aug 22, 2023
5bd3ade
Merge branch 'branch-23.11' into david-destructors-1114
dagardner-nv Aug 24, 2023
07b284a
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Aug 24, 2023
8e4769f
Merge branch 'branch-23.11' into david-pipe-build-refactor
dagardner-nv Aug 29, 2023
080e0c3
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Aug 29, 2023
fab2b98
wip
dagardner-nv Aug 30, 2023
9a00fbe
Ditch the mixin, as sources don't have parent types
dagardner-nv Aug 30, 2023
a4cd8b6
Add output_type method to sources
dagardner-nv Aug 30, 2023
d575aff
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Aug 31, 2023
17a65e8
Mixin for stages which explicitly receive & emit the same type
dagardner-nv Aug 31, 2023
a888e79
Remove unused imports
dagardner-nv Aug 31, 2023
f5149c3
Add an inferred mixin for stages like monitor & trigger
dagardner-nv Aug 31, 2023
d4ea2b5
Helper class to detect union types
dagardner-nv Aug 31, 2023
9043f9e
WIP
dagardner-nv Aug 31, 2023
9228826
WIP
dagardner-nv Aug 31, 2023
d5a9211
WIP
dagardner-nv Aug 31, 2023
14e39ac
WIP
dagardner-nv Aug 31, 2023
a1363be
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Aug 31, 2023
e5a47c1
Define output types for test stages
dagardner-nv Aug 31, 2023
a6b5ef2
Move InMemSourceXStage to its own module to make it easier to find/re…
dagardner-nv Aug 31, 2023
e6c7cbd
WIP
dagardner-nv Aug 31, 2023
81c2ada
CR headers
dagardner-nv Aug 31, 2023
9186a80
wip: 489 tests passing
dagardner-nv Aug 31, 2023
db54e86
wip, 508 tests passing
dagardner-nv Aug 31, 2023
4e8e034
WIP: 1508 tests passing
dagardner-nv Aug 31, 2023
6dc001a
WIP: 1519 tests passing
dagardner-nv Aug 31, 2023
4f0f43b
Set output types for dfp stages, 1706 tests passing
dagardner-nv Aug 31, 2023
e83b803
WIP: 1946 tests passing
dagardner-nv Aug 31, 2023
ff8a478
All tests now passing
dagardner-nv Aug 31, 2023
87f4322
wip
dagardner-nv Aug 31, 2023
efec7ee
Add test to ensure we aren't narrowing types
dagardner-nv Aug 31, 2023
a80fc89
Simplify things a bit
dagardner-nv Aug 31, 2023
4829996
WIP: First attempt at separating the build - untested
dagardner-nv Sep 1, 2023
379ccc2
WIP
dagardner-nv Sep 1, 2023
c916913
WIP
dagardner-nv Sep 1, 2023
07d6834
Fix node/type linking in receiver
dagardner-nv Sep 5, 2023
1e9b7dc
If the pipeline failed any pre-flight checks self._mrc_executor will …
dagardner-nv Sep 5, 2023
b0eee2c
Fix clash between attribute and function
dagardner-nv Sep 5, 2023
132bb34
Fix types for log parsing pipeline
dagardner-nv Sep 5, 2023
d554bcf
Raise when mrc_executor is None
dagardner-nv Sep 5, 2023
e02857b
Add types to stages
dagardner-nv Sep 5, 2023
265a19b
WIP
dagardner-nv Sep 5, 2023
73e8c88
WIP
dagardner-nv Sep 5, 2023
7603d8b
WIP
dagardner-nv Sep 5, 2023
1ee98c1
WIP
dagardner-nv Sep 5, 2023
9ca6efb
WIP
dagardner-nv Sep 5, 2023
7a46d9b
WIP
dagardner-nv Sep 5, 2023
7d37425
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 5, 2023
f2048cd
Fix imports
dagardner-nv Sep 5, 2023
abd3f30
Lint fixes [no ci]
dagardner-nv Sep 5, 2023
17cc476
Move pass_thru.py back [no ci]
dagardner-nv Sep 5, 2023
7768d16
Linting fixes [no ci]
dagardner-nv Sep 5, 2023
b410b0c
Linting fixes [no ci]
dagardner-nv Sep 5, 2023
0e41023
Linting fixes [no ci]
dagardner-nv Sep 5, 2023
b72f2db
Linting fixes [no ci]
dagardner-nv Sep 5, 2023
519be94
Test to ensure we can viz without running (currently not passing) [no…
dagardner-nv Sep 6, 2023
f431100
Visualize no longer requires a full build [no ci]
dagardner-nv Sep 6, 2023
20a5ab2
Verify that we can run after vizualizing [no ci]
dagardner-nv Sep 6, 2023
d967b24
Vizualize prior to running, allowing the viz file to be generated eve…
dagardner-nv Sep 6, 2023
941ff91
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 8, 2023
e95a80b
Add output_type method to RSSSourceStage [no ci]
dagardner-nv Sep 8, 2023
9ffe16b
isort formatting fix
dagardner-nv Sep 8, 2023
5838e6e
formatting [no ci]
dagardner-nv Sep 8, 2023
2c73734
WIP: Stop using streampair in build. TODO: 1) Update *EVERY* _build_s…
dagardner-nv Sep 8, 2023
7929e3e
WIP: Stop using streampair in _post_build. TODO: 1) Update *EVERY* _b…
dagardner-nv Sep 8, 2023
d88a455
WIP: Stop using streampair in _post_build_single. TODO: 1) Update *EV…
dagardner-nv Sep 8, 2023
260b3dc
WIP: Stop using streampair in _build_source. TODO: 1) Update *EVERY* …
dagardner-nv Sep 8, 2023
0a8ff86
SourceStage support multiple outputs . TODO: 1) Update *EVERY* _build…
dagardner-nv Sep 9, 2023
7622532
WIP: Update _build_single, TODO: 1) fix filter_detections_stage, 2) T…
dagardner-nv Sep 11, 2023
c334d93
WIP: fix filter_detections_stage, TODO: 1) remove streampair, 2) Test…
dagardner-nv Sep 11, 2023
e161cf8
Remove import [no ci]
dagardner-nv Sep 11, 2023
bb74ef7
Remove blank lines [no ci]
dagardner-nv Sep 11, 2023
97cc6d1
Remove streampair imports [no ci]
dagardner-nv Sep 11, 2023
9d9085f
No more streampair [no ci]
dagardner-nv Sep 11, 2023
6a04e54
Fix undefined node [no ci]
dagardner-nv Sep 11, 2023
274a3d9
Fix return type [no ci]
dagardner-nv Sep 11, 2023
3e970ad
Fix pre-build check [no ci]
dagardner-nv Sep 11, 2023
5c301b8
Remove unused imports [no ci]
dagardner-nv Sep 11, 2023
7b2f5c1
Add output_type [no ci]
dagardner-nv Sep 11, 2023
4f56150
Update feature length to match value used in tests/test_abp.py
dagardner-nv Sep 12, 2023
7b729dd
Remove TODO as I have tested/verified benchmarks [no ci]
dagardner-nv Sep 12, 2023
b308d82
Update morpheus/pipeline/receiver.py
dagardner-nv Sep 12, 2023
54caf44
Remove unused property method [no ci]
dagardner-nv Sep 12, 2023
293a906
Remove unused property method [no ci]
dagardner-nv Sep 12, 2023
9e6ced9
Update morpheus/pipeline/receiver.py
dagardner-nv Sep 12, 2023
98bcfe6
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 12, 2023
39f62ce
Merge branch 'david-pipe-build-refactor' of github.com:dagardner-nv/M…
dagardner-nv Sep 12, 2023
ff1ddc5
Update morpheus/pipeline/receiver.py
dagardner-nv Sep 12, 2023
dc82070
Method returns plural nodes
dagardner-nv Sep 12, 2023
33ca1ed
Fix type-o [no ci]
dagardner-nv Sep 12, 2023
5b66f9b
Limit the number of loops over the segment_graph are required [no ci]
dagardner-nv Sep 12, 2023
767d5d0
Assert that the pipeline isn't changed after it has been built or pre…
dagardner-nv Sep 12, 2023
dade406
cleanups [no ci]
dagardner-nv Sep 12, 2023
d2469a1
Update morpheus/pipeline/stream_wrapper.py
dagardner-nv Sep 12, 2023
61b0526
Update morpheus/pipeline/stream_wrapper.py
dagardner-nv Sep 12, 2023
c07d002
Update morpheus/pipeline/stream_wrapper.py
dagardner-nv Sep 12, 2023
3f82b47
Update morpheus/pipeline/stream_wrapper.py
dagardner-nv Sep 12, 2023
2ed9f4e
Merge branch 'david-pipe-build-refactor' of github.com:dagardner-nv/M…
dagardner-nv Sep 12, 2023
091ee3c
cleanups [no ci]
dagardner-nv Sep 12, 2023
86c3f40
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 13, 2023
68735a9
Pipeline.pre_build should not be user facing [no ci]
dagardner-nv Sep 13, 2023
f8cdd95
WIP: only call on_start for stages in the current segment [no ci]
dagardner-nv Sep 13, 2023
4bf4b55
Move calling on_start to start_async, and remove the definition of on…
dagardner-nv Sep 13, 2023
dd857c2
Call async_start only on stages in active segments. TODO: this breaks…
dagardner-nv Sep 14, 2023
af33f43
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 18, 2023
e1bcb26
fix calling awaitable coroutine [no ci]
dagardner-nv Sep 19, 2023
1922c0e
Set _is_started and use asyncio.run [no ci]
dagardner-nv Sep 19, 2023
ddba32b
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 19, 2023
8211cc9
Update WriteToElasticsearchStage to new api [no ci]
dagardner-nv Sep 19, 2023
722a75a
Lint fixes [no ci]
dagardner-nv Sep 19, 2023
dea900d
Propagate pre-builds similar to the way builds are propagated. [no ci]
dagardner-nv Sep 20, 2023
8d16e69
Replace /workspace paths with MORPHEUS_ROOT env var, break up long li…
dagardner-nv Sep 21, 2023
8653f56
Fix path to bert vocab file, add missing filter stage, fix line break…
dagardner-nv Sep 21, 2023
062907a
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Sep 27, 2023
0668a7e
Fix merge errors [no ci]
dagardner-nv Sep 27, 2023
176270b
wip
dagardner-nv Sep 29, 2023
3593eb9
wip
dagardner-nv Sep 29, 2023
e788c81
First pass at PortSchema and StageSchema classes
dagardner-nv Sep 29, 2023
b7d1848
Single port helpers
dagardner-nv Sep 29, 2023
de86b9c
wip
dagardner-nv Sep 29, 2023
e5a1002
WIP
dagardner-nv Sep 29, 2023
2978f1f
docstrings
dagardner-nv Oct 2, 2023
16d3f92
WIP
dagardner-nv Oct 2, 2023
679e534
consolidate input schema logic
dagardner-nv Oct 2, 2023
0cdb077
WIP
dagardner-nv Oct 2, 2023
3c56a81
WIP
dagardner-nv Oct 2, 2023
aaa8187
Update the pass-thru-mixin
dagardner-nv Oct 2, 2023
c490ab9
Replace output_type with compute_schema
dagardner-nv Oct 2, 2023
e455b96
Fixing type-os
dagardner-nv Oct 2, 2023
120636b
Fix type-os for viz
dagardner-nv Oct 2, 2023
3ff29d3
Temporarily skip failing test DO NOT MERGE
dagardner-nv Oct 2, 2023
8ae4167
Revert "Temporarily skip failing test DO NOT MERGE"
dagardner-nv Oct 2, 2023
6546644
Complete port schemas produced on the fly by the receiver
dagardner-nv Oct 2, 2023
ff48df3
Lint fixes [no ci]
dagardner-nv Oct 2, 2023
d5a55ac
Rename StreamWrapper to BaseStage to reflect that we are no longer us…
dagardner-nv Oct 5, 2023
5d06a45
WIP: [no ci]
dagardner-nv Oct 5, 2023
64cd7f8
WIP: [no ci]
dagardner-nv Oct 5, 2023
98b1140
Indicate that the as_shared_pointer parameter is unused [no ci]
dagardner-nv Oct 5, 2023
94fb3a7
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 5, 2023
763b88e
Update deltalake stages to support stage schemas [no ci]
dagardner-nv Oct 5, 2023
d57ec20
Remove unused import [no ci]
dagardner-nv Oct 5, 2023
3c0863f
Allow test to run in both Python & C++ mode [no ci]
dagardner-nv Oct 5, 2023
f2803db
Mark the complete method as private [no ci]
dagardner-nv Oct 5, 2023
7cf3c47
Tests for PortSchema [no ci]
dagardner-nv Oct 5, 2023
f0bb070
WIP tests: [no ci]
dagardner-nv Oct 6, 2023
ca72c43
WIP [no ci]
dagardner-nv Oct 6, 2023
bae6e4e
First pass at some mult-port stages for testing [no ci]
dagardner-nv Oct 6, 2023
a7627d1
WIP [no ci]
dagardner-nv Oct 6, 2023
17b7941
WIP tests [no ci]
dagardner-nv Oct 6, 2023
4dcaad3
Move pipeline tests into tests/pipeline
dagardner-nv Oct 6, 2023
9ee5601
Assert that when stages are passed into the add_edge method they only…
dagardner-nv Oct 6, 2023
0aa0f9b
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 6, 2023
65ff2c5
Remove unused imports [no ci]
dagardner-nv Oct 6, 2023
f3a7fa1
Remove TODO comment, will resolve docs in separate PR
dagardner-nv Oct 9, 2023
18438cd
Ignore pylint warning
dagardner-nv Oct 9, 2023
e0a0ea2
Fix import sorting
dagardner-nv Oct 9, 2023
a32ac8f
Merge branch 'branch-23.11' into david-pipe-build-refactor-stage-schema
dagardner-nv Oct 11, 2023
20afe30
Revert unintentional change
dagardner-nv Oct 11, 2023
ea8579a
WIP: Rename BaseStage to StageBase
dagardner-nv Oct 19, 2023
27d9ca5
WIP: Rename BaseStage to StageBase
dagardner-nv Oct 19, 2023
7a57b82
WIP: Rename BaseStage to StageBase
dagardner-nv Oct 19, 2023
dce7991
Merge branch 'branch-23.11' of github.com:nv-morpheus/Morpheus into d…
dagardner-nv Oct 19, 2023
55052e7
Update WriteToVectorDBStage
dagardner-nv Oct 19, 2023
b43f3c7
Fix import order
dagardner-nv Oct 19, 2023
14ce114
Fix import order
dagardner-nv Oct 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
WIP: [no ci]
  • Loading branch information
dagardner-nv committed Oct 5, 2023
commit 5d06a45b494111331aaf0e31c6918d090e2113d1
4 changes: 2 additions & 2 deletions morpheus/pipeline/linear_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -149,8 +149,8 @@ def add_segment_boundary(self, data_type=None, as_shared_pointer=False):
boundary_port_id=self._current_segment_id,
data_type=data_type)

# TODO: update to use data_type once typeid is attached to registered objects out of band:
# https://github.com/nv-morpheus/MRC/issues/176
# TODO: update to use data_type once typeid is attached to registered objects out of band # pylint:disable=fixme
# https://github.com/nv-morpheus/MRC/issues/176
port_id_tuple = (self._current_segment_id, object, False) if data_type else self._current_segment_id

self.add_stage(boundary_egress)
Expand Down
4 changes: 2 additions & 2 deletions morpheus/pipeline/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ def add_edge(self,

Parameters
----------
start : typing.Union[StreamWrapper, Sender]
start : typing.Union[BaseStage, Sender]
The start of the edge or parent stage.

end : typing.Union[Stage, Receiver]
Expand Down Expand Up @@ -265,7 +265,7 @@ def _pre_build(self):
def build(self):
"""
This function sequentially activates all the Morpheus pipeline stages passed by the users to execute a
pipeline. For the `Source` and all added `Stage` objects, `StreamWrapper.build` will be called sequentially to
pipeline. For the `Source` and all added `Stage` objects, `BaseStage.build` will be called sequentially to
construct the pipeline.

Once the pipeline has been constructed, this will start the pipeline by calling `Source.start` on the source
Expand Down
6 changes: 3 additions & 3 deletions morpheus/pipeline/receiver.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,12 @@

class Receiver():
"""
The `Receiver` object represents a downstream port on a `StreamWrapper` object that gets messages from a `Sender`.
The `Receiver` object represents a downstream port on a `BaseStage` object that gets messages from a `Sender`.

Parameters
----------
parent : `morpheus.pipeline.pipeline.StreamWrapper`
Parent `StreamWrapper` object.
parent : `morpheus.pipeline.pipeline.BaseStage`
Parent `BaseStage` object.
port_number : int
Receiver port number.
"""
Expand Down
6 changes: 3 additions & 3 deletions morpheus/pipeline/sender.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@

class Sender():
"""
The `Sender` object represents a port on a `StreamWrapper` object that sends messages to a `Receiver`.
The `Sender` object represents a port on a `BaseStage` object that sends messages to a `Receiver`.

Parameters
----------
parent : `morpheus.pipeline.pipeline.StreamWrapper`
Parent `StreamWrapper` object.
parent : `morpheus.pipeline.pipeline.BaseStage`
Parent `BaseStage` object.
port_number : int
Sender port number.
"""
Expand Down
4 changes: 2 additions & 2 deletions morpheus/pipeline/stream_wrapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ def get_all_input_stages(self) -> list["BaseStage"]:

Returns
-------
list[`morpheus.pipeline.pipeline.StreamWrapper`]
list[`morpheus.pipeline.pipeline.BaseStage`]
All input stages.
"""
return [x.parent for x in self.get_all_inputs()]
Expand All @@ -261,7 +261,7 @@ def get_all_output_stages(self) -> list["BaseStage"]:

Returns
-------
list[`morpheus.pipeline.pipeline.StreamWrapper`]
list[`morpheus.pipeline.pipeline.BaseStage`]
All output stages.
"""
return [x.parent for x in self.get_all_outputs()]
Expand Down