DFP pipeline module #510

bsuryadevara · 2022-11-29T19:13:50Z

Morpheus modules implementation for DFP (Azure, Duo) pipeline training and preprocessing stages
Added modules utility
Moved column_info.py to morpheus.utils
Added tests
Updated digital fingerprinting production example README.md
Added dask and distributed packages
Updated DFP production examples notebooks

…s into dfp-pipeline-module

…e-module

mdemoret-nv

Overall, the code quality looks good. Need a few things addressed regarding naming and commented code before merging. Also would like to discuss how modules should be used with @drobison00 before merging.

docker/conda/environments/requirements.txt

morpheus/modules/file_batcher_module.py

examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_pipeline_preprocessing.py

examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_rolling_window.py

examples/digital_fingerprinting/production/morpheus/dfp/utils/cached_user_window.py

examples/digital_fingerprinting/production/morpheus/dfp_azure_modules_pipeline.py

morpheus/modules/file_to_df.py

…module

mdemoret-nv

Approving. We can merge once the few remaining comments are resolved.

examples/digital_fingerprinting/production/morpheus/dfp/modules/dfp_data_prep.py

examples/digital_fingerprinting/production/morpheus/dfp_azure_modules_pipeline.py

morpheus/modules/file_to_df.py

morpheus/utils/module_ids.py

examples/digital_fingerprinting/production/morpheus/dfp/utils/module_ids.py

…/Morpheus into dfp-pipeline-module

…e-module

mdemoret-nv · 2023-01-24T02:39:32Z

/merge

bsuryadevara added 13 commits November 14, 2022 10:30

dfp modules testing

1427674

Merge branch 'nv-morpheus:branch-22.11' into dfp-training-module

3d769ae

modules stage work

df5066b

Added module stage implementation

ebb2db8

morpehus modules integration

c736828

trivial changes

1d12aee

renamed configuration file

3376ca7

added tests

4253d34

update module factory

b5ded8b

update module factory

4b13b99

Updated linear modules stage

4789d23

Updated linear modules stage

5cdfd68

Merge branch 'nv-morpheus:branch-23.01' into dfp-pipeline-module

d15322a

bsuryadevara added feature request New feature or request 2 - In Progress labels Nov 29, 2022

bsuryadevara self-assigned this Nov 29, 2022

bsuryadevara added 7 commits November 29, 2022 13:18

Updated linear modules stage

bea98a7

Merge branch 'dfp-pipeline-module' of github.com:bsuryadevara/Morpheu…

5a4aa4a

…s into dfp-pipeline-module

renamed mlflow model writer module

f0268b6

added functiontools wraps to a decorator func

25d1c5a

created dfp pipeline preprocessing and training modules

e4a73f2

created dfp pipeline preprocessing and training modules

e5b770f

created dfp pipeline preprocessing and training modules

1575fcd

bsuryadevara added the non-breaking Non-breaking change label Nov 30, 2022

bsuryadevara added 6 commits November 30, 2022 14:15

created dfp pipeline preprocessing and training modules

70fc4cf

used dill to persist source and preprocess schema

5159b0b

used dill to persist source and preprocess schema

9619c0e

renamed files

88d03b1

Updated dfp pipleines with modules

899df5c

Added tests

6a0bcdd

bsuryadevara added 6 commits December 15, 2022 11:46

updated readme, filenames

6e35679

Merge branch 'branch-23.01' into dfp-pipeline-module

7da7189

Updated DFP readme.md

ae9393b

Merge remote-tracking branch 'upstream/branch-23.01' into dfp-pipelin…

4059466

…e-module

moved from srf to mrc

2fc2916

Merge branch 'branch-23.01' into dfp-pipeline-module

62149a7

mdemoret-nv requested changes Jan 17, 2023

View reviewed changes

bsuryadevara added 2 - In Progress and removed 3 - Ready for Review labels Jan 18, 2023

bsuryadevara added this to the Integrated Training and Feedback milestone Jan 18, 2023

bsuryadevara and others added 5 commits January 18, 2023 12:37

changes related to feedback

e0867c4

Merge remote-tracking branch 'origin/branch-23.01' into dfp-pipeline-…

c1af70a

…module

updated header and logging format

0a98249

fix incomplete headers

dd8d69a

Removed commented code

3d5e592

bsuryadevara added 3 - Ready for Review and removed 2 - In Progress labels Jan 19, 2023

mdemoret-nv approved these changes Jan 21, 2023

View reviewed changes

This was referenced Jan 21, 2023

Remove hard-coded keys from production DFP #607

Merged

Fix production DFP pipeline run commands #608

Merged

bsuryadevara and others added 7 commits January 23, 2023 15:13

changed module fucntion name to align with filename

6d30b13

Merge branch 'dfp-pipeline-module' of https://github.com/bsuryadevara…

31538f0

…/Morpheus into dfp-pipeline-module

added distributed dependency to dfp example

b0a42b8

Merge remote-tracking branch 'upstream/branch-23.01' into dfp-pipelin…

904a92e

…e-module

Merge branch-23.01 to dfp-pipeline-module

ba56921

Merge branch 'branch-23.01' into dfp-pipeline-module

6d6e030

resolved merge conflicts

5af5813

ghost merged commit 83479a5 into nv-morpheus:branch-23.01 Jan 24, 2023

bsuryadevara deleted the dfp-pipeline-module branch May 5, 2023 17:50

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DFP pipeline module #510

DFP pipeline module #510

bsuryadevara commented Nov 29, 2022 •

edited

Loading

mdemoret-nv left a comment

mdemoret-nv left a comment

mdemoret-nv commented Jan 24, 2023

DFP pipeline module #510

DFP pipeline module #510

Conversation

bsuryadevara commented Nov 29, 2022 • edited Loading

mdemoret-nv left a comment

Choose a reason for hiding this comment

mdemoret-nv left a comment

Choose a reason for hiding this comment

mdemoret-nv commented Jan 24, 2023

bsuryadevara commented Nov 29, 2022 •

edited

Loading