Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split data dir, moving large files into examples/data #130

Merged
43 commits merged into from
Jun 7, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
61680dc
Add/update unittests to check for issue #60
dagardner-nv Apr 27, 2022
f840074
Ensure default path values are no longer relative to the current dir,…
dagardner-nv Apr 27, 2022
edc75bd
Move simple file reads to a helper function
dagardner-nv May 3, 2022
a7263fc
Merge branch 'branch-22.06' into david-cli-rel-paths
dagardner-nv May 16, 2022
06fb137
WIP
dagardner-nv May 16, 2022
c2c467b
Move data
dagardner-nv May 16, 2022
ce01a4a
Add missing dep for pybind11-stubgen
dagardner-nv May 17, 2022
0b6d959
Don't add deps for pybind11 stub files when we aren't doing an inplac…
dagardner-nv May 17, 2022
827ee41
Add MANIFEST.in to list of installed files
dagardner-nv May 17, 2022
4ef5624
Copy data dir, and files previously set by package_data
dagardner-nv May 17, 2022
c2c5975
Remove package_data, unfortunately the setuptools docs are vague and …
dagardner-nv May 17, 2022
4186357
Remove unused MORPHEUS_ROOT attr
dagardner-nv May 17, 2022
65473c6
Update path in examples for new data location
dagardner-nv May 17, 2022
be44798
Merge branch 'branch-22.06' into david-cli-rel-paths
dagardner-nv May 17, 2022
7ae1e30
Fix import path
dagardner-nv May 17, 2022
329a6a6
Update paths in examples
dagardner-nv May 17, 2022
405b539
Update data path in docs
dagardner-nv May 17, 2022
1c7f421
fix path
dagardner-nv May 17, 2022
c0d5281
Update lfs to reflect data dir move
dagardner-nv May 17, 2022
ce37b33
Remove unneded fea_length
dagardner-nv May 17, 2022
61ebfcf
Style fixes
dagardner-nv May 18, 2022
5a84ff2
Update docs/source/basics/examples.rst
dagardner-nv May 18, 2022
dfdeacc
Merge branch 'branch-22.06' into david-cli-rel-paths
dagardner-nv May 23, 2022
f59dcac
Fixing non-inplace builds install of stub files
mdemoret-nv May 23, 2022
7801803
Move data into previous install command
dagardner-nv May 23, 2022
f398f78
Merge branch 'david-cli-rel-paths' of github.com:dagardner-nv/Morpheu…
dagardner-nv May 23, 2022
798953a
Remove lfs filter for old data location
dagardner-nv May 23, 2022
a94dd62
Merge branch 'branch-22.06' into david-cli-rel-paths
dagardner-nv May 24, 2022
7cfafcf
examples/data/with_data_len.json,examples/data/without_data_len.json:…
dagardner-nv May 27, 2022
950b0d4
Move larger files from morpheus/data into examples/data
dagardner-nv May 27, 2022
4627709
Add new glob path to lfs
dagardner-nv May 27, 2022
0219ae0
Update path in launcher
dagardner-nv May 27, 2022
e32a3c6
Update paths for example data in examples & docs
dagardner-nv May 27, 2022
dac94a9
Add email_with_addresses.jsonlines used in the phishing developer gui…
dagardner-nv May 27, 2022
6448e7f
Merge branch 'branch-22.06' into david-split-data-dir
dagardner-nv May 31, 2022
fc3f06f
Merge branch 'branch-22.06' into david-split-data-dir
dagardner-nv Jun 2, 2022
203c6d6
Remove unused data files
dagardner-nv Jun 3, 2022
e036f7b
Merge branch 'branch-22.06' into david-split-data-dir
dagardner-nv Jun 3, 2022
435c74a
Pin to older neo
dagardner-nv Jun 3, 2022
f13da0b
Merge branch 'david-split-data-dir' of github.com:dagardner-nv/Morphe…
dagardner-nv Jun 3, 2022
5da2d94
Revert "Pin to older neo"
dagardner-nv Jun 6, 2022
d2d35e8
Manually ensure that the build is clean
dagardner-nv Jun 6, 2022
c989b15
Re-source the conda env
dagardner-nv Jun 6, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update path in examples for new data location
  • Loading branch information
dagardner-nv committed May 17, 2022
commit 65473c69a6e710a029ad612fd94cdc710a243f09
4 changes: 2 additions & 2 deletions examples/abp_nvsmi_detection/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ $ nvidia-smi dmon

Each line in the output represents the GPU metrics at a single point in time. As the tool progresses the GPU begins to be utilized and you can see the SM% and Mem% increase as memory is loaded into the GPU and computations are performed. The model we will be using can ingest this information and determine whether or not the GPU is mining cryptocurriences without needing additional information from the host machine.

In this example we will be using the `data/nvsmi.jsonlines` dataset that is known to contain mining behavior profiles. The dataset is in the `.jsonlines` format which means each new line represents an new JSON object. In order to parse this data, it must be ingested, split by lines into individual JSON objects, and parsed into cuDF dataframes. This will all be handled by Morpheus.
In this example we will be using the `morpheus/data/nvsmi.jsonlines` dataset that is known to contain mining behavior profiles. The dataset is in the `.jsonlines` format which means each new line represents an new JSON object. In order to parse this data, it must be ingested, split by lines into individual JSON objects, and parsed into cuDF dataframes. This will all be handled by Morpheus.

## Pipeline Architecture

Expand Down Expand Up @@ -100,7 +100,7 @@ morpheus --log_level=DEBUG \
`# Run a pipeline with 8 threads and a model batch size of 32 (Must be equal or less than Triton config)` \
run --num_threads=8 --pipeline_batch_size=1024 --model_max_batch_size=1024 \
`# Specify a NLP pipeline with 256 sequence length (Must match Triton config)` \
pipeline-fil --columns_file=$MORPHEUS_ROOT/data/columns_fil.txt \
pipeline-fil --model_fea_length 3 \
`# 1st Stage: Read from file` \
from-file --filename=$MORPHEUS_ROOT/data/nvsmi.jsonlines \
`# 2nd Stage: Deserialize from JSON strings to objects` \
Expand Down
2 changes: 1 addition & 1 deletion examples/abp_pcap_detection/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ To launch the configured Morpheus pipeline with the sample data that is provided

```bash
python run.py \
--input_file ../../data/abp_pcap_dump.jsonlines \
--input_file ../../morpheus/data/abp_pcap_dump.jsonlines \
--output_file ./pcap_out.jsonlines \
--model_name 'abp-pcap-xgb' \
--server_url localhost:8001
Expand Down