Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement MultiProcessingStage #1878

Open
wants to merge 28 commits into
base: branch-24.10
Choose a base branch
from
Open

Implement MultiProcessingStage #1878

wants to merge 28 commits into from
+1,164 −0

Conversation

yczhang-nv
Copy link
Contributor

Description

Closes #1850

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@yczhang-nv yczhang-nv requested review from a team as code owners August 29, 2024 23:34
@yczhang-nv yczhang-nv self-assigned this Aug 29, 2024
@yczhang-nv yczhang-nv added non-breaking Non-breaking change feature request New feature or request labels Aug 29, 2024
@yczhang-nv yczhang-nv marked this pull request as draft September 10, 2024 16:45
@yczhang-nv yczhang-nv marked this pull request as ready for review September 18, 2024 19:30
mdemoret-nv
mdemoret-nv requested changes Sep 18, 2024
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very close. Need a few tweaks to how stop/join works. Also, can you add more type annotations to this PR? For any public function, type annotations are helpful.

python/morpheus/morpheus/stages/general/multi_processing_stage.py Outdated Show resolved Hide resolved
python/morpheus/morpheus/stages/general/multi_processing_stage.py Show resolved Hide resolved
python/morpheus/morpheus/stages/general/multi_processing_stage.py Outdated Show resolved Hide resolved
tests/test_multi_processing_stage.py Show resolved Hide resolved
python/morpheus/morpheus/utils/shared_process_pool.py Outdated Show resolved Hide resolved
tests/utils/test_shared_process_pool.py Outdated
Comment on lines 289 to 291
# The pool should be shutdown immediately after calling terminate() without waiting for the tasks to complete
pool.terminate()
pool.join()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are you verifying that the tasks have been shutdown before they are run?

tests/utils/test_shared_process_pool.py Outdated Show resolved Hide resolved
python/morpheus/morpheus/utils/shared_process_pool.py Outdated Show resolved Hide resolved
python/morpheus/morpheus/utils/shared_process_pool.py Outdated Show resolved Hide resolved
python/morpheus/morpheus/utils/shared_process_pool.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@yczhang-nv yczhang-nv

Labels
feature request New feature or request non-breaking Non-breaking change
Projects
Morpheus Boards
Status: Review - Changes Requested
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA]: Add Multi-Processing Stage
2 participants
@yczhang-nv @mdemoret-nv