Skip to content

Latest commit

 

History

History
124 lines (95 loc) · 4.37 KB

examples.rst

File metadata and controls

124 lines (95 loc) · 4.37 KB
Raw

Examples

Simple Identity

This example will copy the values from Kafka into out.jsonlines.

img/simple_identity.png

morpheus run pipeline-nlp --viz_file=basic_usage_img/simple_identity.png  \
   from-kafka --input_topic test_pcap \
   deserialize \
   serialize \
   to-file --filename out.jsonlines

Remove Fields from JSON Objects

This example will only copy the fields 'timestamp', 'src_ip' and 'dest_ip' from examples/data/pcap_dump.jsonlines to out.jsonlines.

img/remove_fields_from_json_objects.png

morpheus run pipeline-nlp --viz_file=basic_usage_img/remove_fields_from_json_objects.png \
   from-file --filename examples/data/pcap_dump.jsonlines \
   deserialize \
   serialize --include 'timestamp' --include 'src_ip' --include 'dest_ip' \
   to-file --filename out.jsonlines

Monitor Throughput

This example will report the throughput on the command line.

img/monitor_throughput.png

$ morpheus run pipeline-nlp --viz_file=basic_usage_img/monitor_throughput.png  \
   from-file --filename examples/data/pcap_dump.jsonlines \
   deserialize \
   monitor --description "Lines Throughput" --smoothing 0.1 --unit "lines" \
   serialize \
   to-file --filename out.jsonlines
Configuring Pipeline via CLI
Starting pipeline via CLI... Ctrl+C to Quit
Pipeline visualization saved to basic_usage_img/monitor_throughput.png
Lines Throughput: 88064lines [00:11, 7529.37lines/s]

Multi-Monitor Throughput

This example will report the throughput for each stage independently. Keep in mind, buffer stages are necessary to decouple one stage from the next. Without the buffers, all montioring would show the same throughput.

img/multi_monitor_throughput.png

$ morpheus run pipeline-nlp --viz_file=basic_usage_img/multi_monitor_throughput.png  \
   from-file --filename examples/data/pcap_dump.jsonlines \
   monitor --description "From File Throughput" \
   buffer \
   deserialize \
   monitor --description "Deserialize Throughput" \
   buffer \
   serialize \
   monitor --description "Serialize Throughput" \
   buffer \
   to-file --filename out.jsonlines --overwrite
Configuring Pipeline via CLI
Starting pipeline via CLI... Ctrl+C to Quit
Pipeline visualization saved to basic_usage_img/multi_monitor_throughput.png
From File Throughput: 93085messages [00:09, 83515.94messages/s]
Deserialize Throughput: 93085messages [00:20, 9783.56messages/s]
Serialize Throughput: 93085messages [00:20, 9782.07messages/s]

NLP Kitchen Sink

This example shows an NLP Pipeline which uses most stages available in Morpheus.

img/nlp_kitchen_sink.png

$ morpheus run --num_threads=8 --pipeline_batch_size=1024 --model_max_batch_size=32 \
   pipeline-nlp --viz_file=basic_usage_img/nlp_kitchen_sink.png  \
   from-file --filename examples/data/pcap_dump.jsonlines \
   buffer --count=500 \
   deserialize \
   preprocess \
   buffer \
   inf-triton --model_name=sid-minibert-onnx --server_url=localhost:8001 \
   monitor --description "Inference Rate" --smoothing=0.001 --unit "inf" \
   add-class \
   filter --threshold=0.8 \
   serialize --include 'timestamp' --exclude '^_ts_' \
   to-kafka --output_topic "inference_output"
Configuring Pipeline via CLI
Starting pipeline via CLI... Ctrl+C to Quit
Pipeline visualization saved to basic_usage_img/nlp_kitchen_sink.png
Inference Rate: 16384inf [19:50, 13.83inf/s]