Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DFP pipeline module #510

Merged
64 commits merged into from
Jan 24, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
1427674
dfp modules testing
bsuryadevara Nov 14, 2022
3d769ae
Merge branch 'nv-morpheus:branch-22.11' into dfp-training-module
bsuryadevara Nov 14, 2022
df5066b
modules stage work
bsuryadevara Nov 18, 2022
ebb2db8
Added module stage implementation
bsuryadevara Nov 22, 2022
c736828
morpehus modules integration
bsuryadevara Nov 22, 2022
1d12aee
trivial changes
bsuryadevara Nov 22, 2022
3376ca7
renamed configuration file
bsuryadevara Nov 23, 2022
4253d34
added tests
bsuryadevara Nov 28, 2022
b5ded8b
update module factory
bsuryadevara Nov 28, 2022
4b13b99
update module factory
bsuryadevara Nov 28, 2022
4789d23
Updated linear modules stage
bsuryadevara Nov 29, 2022
5cdfd68
Updated linear modules stage
bsuryadevara Nov 29, 2022
d15322a
Merge branch 'nv-morpheus:branch-23.01' into dfp-pipeline-module
bsuryadevara Nov 29, 2022
bea98a7
Updated linear modules stage
bsuryadevara Nov 29, 2022
5a4aa4a
Merge branch 'dfp-pipeline-module' of github.com:bsuryadevara/Morpheu…
bsuryadevara Nov 29, 2022
f0268b6
renamed mlflow model writer module
bsuryadevara Nov 29, 2022
25d1c5a
added functiontools wraps to a decorator func
bsuryadevara Nov 29, 2022
e4a73f2
created dfp pipeline preprocessing and training modules
bsuryadevara Nov 30, 2022
e5b770f
created dfp pipeline preprocessing and training modules
bsuryadevara Nov 30, 2022
1575fcd
created dfp pipeline preprocessing and training modules
bsuryadevara Nov 30, 2022
70fc4cf
created dfp pipeline preprocessing and training modules
bsuryadevara Nov 30, 2022
5159b0b
used dill to persist source and preprocess schema
bsuryadevara Nov 30, 2022
9619c0e
used dill to persist source and preprocess schema
bsuryadevara Nov 30, 2022
88d03b1
renamed files
bsuryadevara Dec 1, 2022
899df5c
Updated dfp pipleines with modules
bsuryadevara Dec 7, 2022
6a0bcdd
Added tests
bsuryadevara Dec 7, 2022
e85ee77
resolved merge conflicts
bsuryadevara Dec 7, 2022
dd9f2d2
created dfp (azure, duo) training pipelines with modules
bsuryadevara Dec 7, 2022
0bacf6e
style correction
bsuryadevara Dec 7, 2022
d9983dd
style correction
bsuryadevara Dec 7, 2022
2185886
style correction
bsuryadevara Dec 8, 2022
e50d1d1
added dask and distributed packages to requirements.txt
bsuryadevara Dec 8, 2022
4fdd4c9
Merge branch 'branch-23.01' into dfp-pipeline-module
bsuryadevara Dec 8, 2022
0e6d5cf
Resloved conflicts
bsuryadevara Dec 8, 2022
9158130
addedd missing tests file
bsuryadevara Dec 9, 2022
e0e7309
Merge remote-tracking branch 'upstream/branch-23.01' into dfp-pipelin…
bsuryadevara Dec 9, 2022
c91aa7f
fix to failing test
bsuryadevara Dec 10, 2022
a9d3e9e
fix to failing test
bsuryadevara Dec 10, 2022
516e746
fix tests
bsuryadevara Dec 10, 2022
bdf384a
addressed feedback comments
bsuryadevara Dec 13, 2022
c6c49a9
Update examples/digital_fingerprinting/production/morpheus/dfp_azure_…
bsuryadevara Dec 13, 2022
c93ac48
Update examples/digital_fingerprinting/production/morpheus/dfp_duo_mo…
bsuryadevara Dec 13, 2022
7e30dea
input and output port names from a module as params
bsuryadevara Dec 13, 2022
8aa84f5
Merge branch 'branch-23.01' into dfp-pipeline-module
bsuryadevara Dec 13, 2022
2fb7a5e
Update morpheus/stages/general/linear_modules_stage.py
bsuryadevara Dec 14, 2022
a4ce99e
Merge branch 'branch-23.01' into dfp-pipeline-module
bsuryadevara Dec 14, 2022
6e35679
updated readme, filenames
bsuryadevara Dec 15, 2022
7da7189
Merge branch 'branch-23.01' into dfp-pipeline-module
bsuryadevara Dec 15, 2022
ae9393b
Updated DFP readme.md
bsuryadevara Dec 15, 2022
4059466
Merge remote-tracking branch 'upstream/branch-23.01' into dfp-pipelin…
bsuryadevara Dec 20, 2022
2fc2916
moved from srf to mrc
bsuryadevara Dec 20, 2022
62149a7
Merge branch 'branch-23.01' into dfp-pipeline-module
bsuryadevara Jan 4, 2023
e0867c4
changes related to feedback
bsuryadevara Jan 18, 2023
c1af70a
Merge remote-tracking branch 'origin/branch-23.01' into dfp-pipeline-…
bsuryadevara Jan 18, 2023
0a98249
updated header and logging format
bsuryadevara Jan 18, 2023
dd8d69a
fix incomplete headers
bsuryadevara Jan 18, 2023
3d5e592
Removed commented code
bsuryadevara Jan 19, 2023
6d30b13
changed module fucntion name to align with filename
bsuryadevara Jan 23, 2023
31538f0
Merge branch 'dfp-pipeline-module' of https://github.com/bsuryadevara…
bsuryadevara Jan 23, 2023
b0a42b8
added distributed dependency to dfp example
bsuryadevara Jan 23, 2023
904a92e
Merge remote-tracking branch 'upstream/branch-23.01' into dfp-pipelin…
bsuryadevara Jan 23, 2023
ba56921
Merge branch-23.01 to dfp-pipeline-module
bsuryadevara Jan 23, 2023
6d6e030
Merge branch 'branch-23.01' into dfp-pipeline-module
bsuryadevara Jan 23, 2023
5af5813
resolved merge conflicts
bsuryadevara Jan 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
changed module fucntion name to align with filename
  • Loading branch information
bsuryadevara committed Jan 23, 2023
commit 6d30b13182ea0c9f8cc95c5a62c388ae25a95acd
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@


@register_module(DFP_DATA_PREP, MODULE_NAMESPACE)
def dfp_preprocessing(builder: mrc.Builder):
def dfp_data_prep(builder: mrc.Builder):
"""
Preprocessed data are produced by this module function for either inference or model training.
This module function prepares data for either inference or model training.

Parameters
----------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import morpheus.modules.file_batcher # noqa: F401
import morpheus.modules.file_to_df # noqa: F401
from morpheus.utils.module_ids import FILE_BATCHER
from morpheus.utils.module_ids import FILE_TO_DATAFRAME
from morpheus.utils.module_ids import FILE_TO_DF
from morpheus.utils.module_ids import MODULE_NAMESPACE
from morpheus.utils.module_utils import get_module_config
from morpheus.utils.module_utils import load_module
Expand All @@ -38,7 +38,7 @@


@register_module(DFP_PREPROCESSING, MODULE_NAMESPACE)
def dfp_pipeline_preprocessing(builder: mrc.Builder):
def dfp_preprocessing(builder: mrc.Builder):
bsuryadevara marked this conversation as resolved.
Show resolved Hide resolved
"""
This module function allows for the consolidation of multiple dfp pipeline preprocessing modules
into a single module.
Expand All @@ -52,7 +52,7 @@ def dfp_pipeline_preprocessing(builder: mrc.Builder):
config = get_module_config(DFP_PREPROCESSING, builder)

file_batcher_conf = config.get(FILE_BATCHER, None)
file_to_df_conf = config.get(FILE_TO_DATAFRAME, None)
file_to_df_conf = config.get(FILE_TO_DF, None)
dfp_split_users_conf = config.get(DFP_SPLIT_USERS, None)
dfp_rolling_window_conf = config.get(DFP_ROLLING_WINDOW, None)
dfp_data_prep_conf = config.get(DFP_DATA_PREP, None)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@
from morpheus.utils.logger import get_log_levels
from morpheus.utils.logger import parse_log_level
from morpheus.utils.module_ids import FILE_BATCHER
from morpheus.utils.module_ids import FILE_TO_DATAFRAME
from morpheus.utils.module_ids import FILE_TO_DF
from morpheus.utils.module_ids import MLFLOW_MODEL_WRITER
from morpheus.utils.module_ids import MODULE_NAMESPACE

Expand Down Expand Up @@ -261,9 +261,9 @@ def run_pipeline(train_users,
"end_time": end_time,
"iso_date_regex_pattern": iso_date_regex_pattern
},
FILE_TO_DATAFRAME: {
"module_id": FILE_TO_DATAFRAME,
"module_name": "file_to_dataframe",
FILE_TO_DF: {
"module_id": FILE_TO_DF,
"module_name": "FILE_TO_DF",
"namespace": MODULE_NAMESPACE,
"timestamp_column_name": config.ae.timestamp_column_name,
"userid_column_name": config.ae.userid_column_name,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
from morpheus.utils.logger import get_log_levels
from morpheus.utils.logger import parse_log_level
from morpheus.utils.module_ids import FILE_BATCHER
from morpheus.utils.module_ids import FILE_TO_DATAFRAME
from morpheus.utils.module_ids import FILE_TO_DF
from morpheus.utils.module_ids import MLFLOW_MODEL_WRITER
from morpheus.utils.module_ids import MODULE_NAMESPACE

Expand Down Expand Up @@ -259,9 +259,9 @@ def run_pipeline(train_users,
"end_time": end_time,
"iso_date_regex_pattern": iso_date_regex_pattern
},
FILE_TO_DATAFRAME: {
"module_id": FILE_TO_DATAFRAME,
"module_name": "file_to_dataframe",
FILE_TO_DF: {
"module_id": FILE_TO_DF,
"module_name": "FILE_TO_DF",
"namespace": MODULE_NAMESPACE,
"timestamp_column_name": config.ae.timestamp_column_name,
"userid_column_name": config.ae.userid_column_name,
Expand Down
10 changes: 5 additions & 5 deletions morpheus/modules/file_to_df.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@
from morpheus.cli.utils import str_to_file_type
from morpheus.io.deserializers import read_file_to_df
from morpheus.utils.column_info import process_dataframe
from morpheus.utils.module_ids import FILE_TO_DATAFRAME
from morpheus.utils.module_ids import FILE_TO_DF
from morpheus.utils.module_ids import MODULE_NAMESPACE
from morpheus.utils.module_utils import get_module_config
from morpheus.utils.module_utils import register_module

logger = logging.getLogger(__name__)


@register_module(FILE_TO_DATAFRAME, MODULE_NAMESPACE)
def file_to_dataframe(builder: mrc.Builder):
@register_module(FILE_TO_DF, MODULE_NAMESPACE)
def file_to_df(builder: mrc.Builder):
"""
This module reads data from the batched files into a dataframe after receiving input from the "FileBatcher" module.
In addition to loading data from the disk, it has ability to load the file content from S3 buckets.
Expand All @@ -54,7 +54,7 @@ def file_to_dataframe(builder: mrc.Builder):
mrc Builder object.
"""

config = get_module_config(FILE_TO_DATAFRAME, builder)
config = get_module_config(FILE_TO_DF, builder)

schema_config = config.get("schema", None)
schema_str = schema_config.get("schema_str", None)
Expand Down Expand Up @@ -254,7 +254,7 @@ def node_fn(obs: mrc.Observable, sub: mrc.Subscriber):
if (download_method.startswith("dask")):
dask_cluster = get_dask_cluster()

node = builder.make_node_full(FILE_TO_DATAFRAME, node_fn)
node = builder.make_node_full(FILE_TO_DF, node_fn)

# Register input and output port for a module.
builder.register_module_input("input", node)
Expand Down
2 changes: 1 addition & 1 deletion morpheus/modules/mlflow_model_writer.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@


@register_module(MLFLOW_MODEL_WRITER, MODULE_NAMESPACE)
def module_init(builder: mrc.Builder):
def mlflow_model_writer(builder: mrc.Builder):
"""
This module uploads trained models to the mlflow server.

Expand Down
2 changes: 1 addition & 1 deletion morpheus/utils/module_ids.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,5 +15,5 @@
MODULE_NAMESPACE = "morpheus_modules"

FILE_BATCHER = "FileBatcher"
FILE_TO_DATAFRAME = "FileToDataFrame"
FILE_TO_DF = "FileToDataFrame"
MLFLOW_MODEL_WRITER = "MLFlowModelWriter"