Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tests for stages in DFP Production Example #940

Merged
merged 304 commits into from
May 18, 2023
Merged

Tests for stages in DFP Production Example #940

merged 304 commits into from
May 18, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented May 5, 2023 •
edited
Loading

Description

  • Adds a test for each of the stages in the DFP production example (tests for modules coming in separate PR).
  • Add/fix type-hints to existing stages where missing
  • Fix type-o in DFPFileToDataFrameStage. The type-hint and documentation specifies valid values for MORPHEUS_FILE_DOWNLOAD_TYPE as ["single_thread", "multiprocess", "dask", "dask_thread"] however in practice "multiprocessing" is the value used. Updated to accept either value.
  • Remove unused method _trim_dataframe from DFPRollingWindowStage this logic appears to have been moved to CachedUserWindow.trim_dataframe
  • Fix accepted types for DFPVizPostprocStage
  • Move nested write_to_files method in DFPVizPostprocStage to an instance method allowing it to be tested.
  • Fix format detection logic in read_file_to_df when FileTypes=Auto and an instance of an fsspec opener was passed in instead of the expected file path.
  • Consolidate duplicated DataFrame downloading code
  • Fix bug in PreallocatorMixin when the incoming datatype is a DataFrame and not a message

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

dagardner-nv and others added 30 commits April 13, 2023 14:12
@dagardner-nv
Merge branch 'branch-23.07' into david-test-examples
610b288
@dagardner-nv
Merge branch 'branch-23.07' into david-test-examples
517b964
@dagardner-nv
Optionally bypass the cache and optionally pass additional kwargs
a3aef0a
@dagardner-nv
Fix usage of assert_df_equal which is now a static method of DatasetM…
d2065ad 
…anager
@dagardner-nv
Use datasetmanger
b42d7c8
@dagardner-nv
Use new dataset fixtures [no ci]
de73c01
@dagardner-nv
Revert code updates due to removing redundant bert files, will replac…
3cb8b98 
…e those with symlinks for compat [no ci]
@dagardner-nv
Add bert vocab files to morpheus/data to be available for testing, pr…
1215fbe 
…eviously these were in LFS in stored under models/training-tuning-scripts/sid-models/resources/
@dagardner-nv
Updates due to moving the bert vocabs [no ci]
ce57304
@dagardner-nv
wip [no ci]
e990cbf
@dagardner-nv
Leave symlinks from the old locations for bert data to the new locati…
2e97a48 
…on for compatibility
@dagardner-nv
Remove unused import
049c5d5
@dagardner-nv
Shorted comment to fix under 100 column limit
fc91053
@dagardner-nv
Move example tests data to its own subfolder
225bf9f
@dagardner-nv
Update tests to reflect data move [no ci]
7fa07b5
@dagardner-nv
Copy of the first 10 lines of examples/data/email_with_addresses.json…
e4460d2 
…lines since examples/data is quite large and not fetched in CI
@dagardner-nv
Update test to reflect data move [no ci]
7b9c440
@dagardner-nv
Merge branch 'branch-23.07' of github.com:nv-morpheus/Morpheus into d…
fc8edeb 
…avid-test-examples [no ci]
@dagardner-nv
Add shared libs from examples to tar
39f4f3b
@dagardner-nv
Merge branch 'branch-23.07' of github.com:nv-morpheus/Morpheus into d…
fea441f 
…avid-test-examples
@dagardner-nv
Add the shared libs from examples not the build dir
e45508f
@dagardner-nv
Remove work-around for https://docs.rapids.ai/notices/rsn0026/
7dafb10
@dagardner-nv
remove out-dated protobuf version deps
8a72abe
@dagardner-nv
Remove unused anaconda channel, no longer needed to pull in a specifi…
3993573 
…c libprotobuf version
@dagardner-nv
Use mamba not conda
8d520a8
@dagardner-nv
Remove constrained versions which are no longer needed for our curren…
f66a12c 
…t version of rapids
@dagardner-nv
Move the yaml for the gnn fraud example to the docker/conda/environme…
caaab97 
…nts dir, and replace it with a symlink for compatibility
@dagardner-nv
Add explicit deps for dask and distributed
18490a9
@dagardner-nv
Don't install node&npm from apt as we install it with conda, add extr…
b13290b 
…a deps needed for examples to the test target
@dagardner-nv
Merge branch 'david-test-examples' into david-test-examples-p2
443c8e9
@dagardner-nv dagardner-nv added 3 - Ready for Review and removed 2 - In Progress Merge After Dependencies PR is completed and reviewed but depends on another PR; do not merge out of order labels May 17, 2023
mdemoret-nv
mdemoret-nv approved these changes May 17, 2023
View reviewed changes
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge when comments are addressed

examples/digital_fingerprinting/production/morpheus/dfp/stages/dfp_file_to_df.py Outdated Show resolved Hide resolved
morpheus/pipeline/preallocator_mixin.py Outdated Show resolved Hide resolved
morpheus/utils/downloader.py Outdated Show resolved Hide resolved
tests/examples/digital_fingerprinting/conftest.py Outdated Show resolved Hide resolved
tests/examples/digital_fingerprinting/test_dfp_file_batcher_stage.py Outdated Show resolved Hide resolved
tests/examples/digital_fingerprinting/test_dfp_file_batcher_stage.py Outdated Show resolved Hide resolved
tests/examples/digital_fingerprinting/test_dfp_file_to_df.py Outdated Show resolved Hide resolved
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit c703615 into nv-morpheus:branch-23.07 May 18, 2023
@dagardner-nv dagardner-nv deleted the david-test-dfp-examples branch May 18, 2023 18:16
@dagardner-nv dagardner-nv mentioned this pull request Jun 13, 2023
Closed
2 tasks
@mdemoret-nv mdemoret-nv mentioned this pull request Aug 21, 2023
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dagardner-nv @mdemoret-nv