Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use SRF Logging #266

Merged
5 commits merged into from
Jul 26, 2022
Merged

Conversation

mdemoret-nv
Copy link
Contributor

@mdemoret-nv mdemoret-nv commented Jul 6, 2022

Addresses #137

Closes #306

@mdemoret-nv mdemoret-nv added non-breaking Non-breaking change feature request New feature or request 2 - In Progress labels Jul 6, 2022
@mdemoret-nv mdemoret-nv requested a review from a team as a code owner July 6, 2022 17:04
@mdemoret-nv
Copy link
Contributor Author

Not sure how I feel about this PR now. Using the SRF logger (i.e. GLOG), causes the output to be difficult to read since the prefixes are not constant length. The output log looks like the following:

I20220726 10:19:31.925791 27421 cli.py:108] Parameter, 'labels_file', with relative path, 'data/labels_nlp.txt', does not exist. Using package relative location: '/home/mdemoret/Repos/morpheus/morpheus-dev/morpheus/data/labels_nlp.txt'
I20220726 10:20:08.877607 27421 cli.py:408] Loaded labels file. Current labels: [['address', 'bank_acct', 'credit_card', 'email', 'govt_id', 'name', 'password', 'phone_num', 'secret_keys', 'user']]
Starting pipeline via CLI... Ctrl+C to Quit
I20220726 10:20:14.705170 27421 cli.py:587] Config: 
{
  "ae": null,
  "class_labels": [
    "address",
    "bank_acct",
    "credit_card",
    "email",
    "govt_id",
    "name",
    "password",
    "phone_num",
    "secret_keys",
    "user"
  ],
  "debug": false,
  "edge_buffer_size": 4,
  "feature_length": 256,
  "fil": null,
  "log_config_file": null,
  "log_level": 10,
  "mode": "NLP",
  "model_max_batch_size": 32,
  "num_threads": 8,
  "pipeline_batch_size": 1024
}
I20220726 10:20:14.706020 27421 cli.py:588] CPP Enabled: False
I20220726 10:20:14.706730 27421 pipeline.py:143] ====Registering Pipeline====
I20220726 10:20:14.732419 27421 pipeline.py:191] ====Registering Pipeline Complete!====
I20220726 10:20:14.737983 27421 pipeline.py:196] ====Starting Pipeline====
I20220726 10:20:14.740716 27421 pipeline.py:200] ====Pipeline Started====
I20220726 10:20:14.742210 27421 pipeline.py:150] ====Building Pipeline====
I20220726 10:20:14.743883 27421 single_output_source.py:51] Added source: <from-file-0; FileSourceStage(filename=models/datasets/validation-data/sid-validation-data.csv, iterative=False, file_type=FileTypes.Auto, repeat=50, filter_null=True, cudf_kwargs=None)>
  └─> morpheus.MessageMeta
I20220726 10:20:14.745198 27421 single_port_stage.py:94] Added stage: <deserialize-1; DeserializeStage()>
  └─ morpheus.MessageMeta -> morpheus.MultiMessage
I20220726 10:20:14.746296 27421 single_port_stage.py:94] Added stage: <preprocess-nlp-2; PreprocessNLPStage(vocab_hash_file=/home/mdemoret/Repos/morpheus/morpheus-dev/morpheus/data/bert-base-uncased-hash.txt, truncation=True, do_lower_case=True, add_special_tokens=False, stride=-1)>
  └─ morpheus.MultiMessage -> morpheus.MultiInferenceNLPMessage
I20220726 10:20:14.747447 27421 single_port_stage.py:94] Added stage: <inference-3; TritonInferenceStage(model_name=sid-minibert-onnx, server_url=localhost:8001, force_convert_inputs=True, use_shared_memory=False)>
  └─ morpheus.MultiInferenceNLPMessage -> morpheus.MultiResponseProbsMessage
I20220726 10:20:14.748368 27421 single_port_stage.py:94] Added stage: <monitor-4; MonitorStage(description=Inference Rate, smoothing=0.05, unit=inf, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
I20220726 10:20:14.749765 27421 single_port_stage.py:94] Added stage: <add-class-5; AddClassificationsStage(threshold=0.5, labels=[], prefix=)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
I20220726 10:20:14.751262 27421 single_port_stage.py:94] Added stage: <serialize-6; SerializeStage(include=[], exclude=['^_ts_'], fixed_columns=True)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
I20220726 10:20:14.752252 27421 single_port_stage.py:94] Added stage: <to-file-7; WriteToFileStage(filename=./.tmp/sid_training_data-nlp.csv, overwrite=True, file_type=FileTypes.Auto)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
I20220726 10:20:14.753262 27421 pipeline.py:179] ====Building Pipeline Complete!====
I20220726 10:20:14.754415 27421 pipeline.py:249] Starting! Time: 1658852414.7363098

This is what it looks like currently:

Configuring Pipeline via CLI
Loaded labels file. Current labels: [['address', 'bank_acct', 'credit_card', 'email', 'govt_id', 'name', 'password', 'phone_num', 'secret_keys', 'user']]
Starting pipeline via CLI... Ctrl+C to Quit
Config: 
{
  "ae": null,
  "class_labels": [
    "address",
    "bank_acct",
    "credit_card",
    "email",
    "govt_id",
    "name",
    "password",
    "phone_num",
    "secret_keys",
    "user"
  ],
  "debug": false,
  "edge_buffer_size": 4,
  "feature_length": 256,
  "fil": null,
  "log_config_file": null,
  "log_level": 10,
  "mode": "NLP",
  "model_max_batch_size": 32,
  "num_threads": 8,
  "pipeline_batch_size": 1024
}
CPP Enabled: False
====Registering Pipeline====
====Registering Pipeline Complete!====
====Starting Pipeline====
====Pipeline Started====              
====Building Pipeline====             
Added source: <from-file-0; FileSourceStage(filename=models/datasets/validation-data/sid-validation-data.csv, iterative=False, file_type=FileTypes.Auto, repeat=50, filter_null=True, cudf_kwargs=None)>
  └─> morpheus.MessageMeta
Added stage: <deserialize-1; DeserializeStage()>
  └─ morpheus.MessageMeta -> morpheus.MultiMessage
Added stage: <preprocess-nlp-2; PreprocessNLPStage(vocab_hash_file=/home/mdemoret/Repos/morpheus/morpheus-dev/morpheus/data/bert-base-uncased-hash.txt, truncation=True, do_lower_case=True, add_special_tokens=False, stride=-1)>
  └─ morpheus.MultiMessage -> morpheus.MultiInferenceNLPMessage
Added stage: <inference-3; TritonInferenceStage(model_name=sid-minibert-onnx, server_url=localhost:8001, force_convert_inputs=True, use_shared_memory=False)>
  └─ morpheus.MultiInferenceNLPMessage -> morpheus.MultiResponseProbsMessage
Added stage: <monitor-4; MonitorStage(description=Inference Rate, smoothing=0.05, unit=inf, delayed_start=False, determine_count_fn=None)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <add-class-5; AddClassificationsStage(threshold=0.5, labels=[], prefix=)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MultiResponseProbsMessage
Added stage: <serialize-6; SerializeStage(include=[], exclude=['^_ts_'], fixed_columns=True)>
  └─ morpheus.MultiResponseProbsMessage -> morpheus.MessageMeta
Added stage: <to-file-7; WriteToFileStage(filename=./.tmp/sid_training_data-nlp.csv, overwrite=True, file_type=FileTypes.Auto)>
  └─ morpheus.MessageMeta -> morpheus.MessageMeta
====Building Pipeline Complete!====   
Starting! Time: 1658853060.4051416

@mdemoret-nv
Copy link
Contributor Author

mdemoret-nv commented Jul 26, 2022

Breaking this up into 2 steps:

  1. Initialize the SRF logger to prevent warning messages and have a consistent log level ([BUG] Initialize SRF Logger #306)
  2. Use the SRF logger in TqdmLoggingHandler ([FEA] Use SRF's log handler #137)

This PR will only close #306

@mdemoret-nv
Copy link
Contributor Author

@gpucibot merge

@ghost ghost merged commit b04392f into nv-morpheus:branch-22.08 Jul 26, 2022
@mdemoret-nv mdemoret-nv deleted the mdd_use-srf-logging branch July 26, 2022 23:26
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[BUG] Initialize SRF Logger
2 participants