Skip to content

Commit

Permalink
Documentation updates (#519)
Browse files Browse the repository at this point in the history
- Include markdown documentation under the `examples` dir in Sphinx documentation builds by creating symlinks in `docs/source` to the md files in the examples dir.
- Replace some ableist language appearing in the documentation.
- Remove explcit calls to pipeline.build, this is no longer needed
- Update github url for Morpheus

relates to issue #495

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)
  - https://github.com/raykallen

URL: #519
  • Loading branch information
dagardner-nv authored Dec 8, 2022
1 parent e681951 commit 1fec0a5
Show file tree
Hide file tree
Showing 42 changed files with 38,309 additions and 143 deletions.
14 changes: 7 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
Contributions to Morpheus fall into the following three categories.

1. To report a bug, request a new feature, or report a problem with
documentation, please file an [issue](https://github.com/NVIDIA/Morpheus/issues/new)
documentation, please file an [issue](https://github.com/nv-morpheus/Morpheus/issues/new)
describing in detail the problem or new feature. The Morpheus team evaluates
and triages issues, and schedules them for a release. If you believe the
issue needs priority attention, please comment on the issue to notify the
team.
2. To propose and implement a new Feature, please file a new feature request
[issue](https://github.com/NVIDIA/Morpheus/issues/new). Describe the
[issue](https://github.com/nv-morpheus/Morpheus/issues/new). Describe the
intended feature and discuss the design and implementation with the team and
community. Once the team agrees that the plan looks good, go ahead and
implement it, using the [code contributions](#code-contributions) guide below.
Expand All @@ -25,18 +25,18 @@ More information can be found at: [Contributor Code of Conduct](CODE_OF_CONDUCT.

### Your first issue

1. Find an issue to work on. The best way is to look for issues with the [good first issue](https://github.com/NVIDIA/Morpheus/issues) label.
1. Find an issue to work on. The best way is to look for issues with the [good first issue](https://github.com/nv-morpheus/Morpheus/issues) label.
2. Comment on the issue stating that you are going to work on it.
3. Code! Make sure to update unit tests! Ensure the [license headers are set properly](#Licensing).
4. When done, [create your pull request](https://github.com/NVIDIA/Morpheus/compare).
4. When done, [create your pull request](https://github.com/nv-morpheus/Morpheus/compare).
5. Wait for other developers to review your code and update code as needed.
6. Once reviewed and approved, a Morpheus developer will merge your pull request.

Remember, if you are unsure about anything, don't hesitate to comment on issues and ask for clarifications!

### Seasoned developers

Once you have gotten your feet wet and are more comfortable with the code, you can look at the prioritized issues for our next release in our [project boards](https://github.com/NVIDIA/Morpheus/projects).
Once you have gotten your feet wet and are more comfortable with the code, you can look at the prioritized issues for our next release in our [project boards](https://github.com/nv-morpheus/Morpheus/projects).

> **Pro Tip:** Always look at the release board with the highest number for issues to work on. This is where Morpheus developers also focus their efforts.
Expand All @@ -60,7 +60,7 @@ export PYTHON_VER=3.8
export RAPIDS_VER=22.08
export CUDA_VER=11.5
export MORPHEUS_ROOT=$(pwd)/morpheus
git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
git clone https://github.com/nv-morpheus/Morpheus.git $MORPHEUS_ROOT
cd $MORPHEUS_ROOT
```
The large model and data files in this repo are stored using [Git Large File Storage (LFS)](https://git-lfs.github.com/). These files will be required for running the training/validation scripts and example pipelines for the Morpheus pre-trained models.
Expand Down Expand Up @@ -168,7 +168,7 @@ Note: These instructions assume the user is using `mamba` instead of `conda` sin
export RAPIDS_VER=22.08
export CUDA_VER=11.5
export MORPHEUS_ROOT=$(pwd)/morpheus
git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
git clone https://github.com/nv-morpheus/Morpheus.git $MORPHEUS_ROOT
cd $MORPHEUS_ROOT
```
1. Create the morpheus Conda environment
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The following sections must be followed prior to building the Morpheus container

```bash
MORPHEUS_ROOT=$(pwd)/morpheus
git clone https://github.com/NVIDIA/Morpheus.git $MORPHEUS_ROOT
git clone https://github.com/nv-morpheus/Morpheus.git $MORPHEUS_ROOT
cd $MORPHEUS_ROOT
```

Expand Down
8 changes: 4 additions & 4 deletions docs/source/basics/building_a_pipeline.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,12 @@ All pipelines configured with the CLI need to start with a source object. Curren
* ``from-kafka``
- Pulls messages from a Kafka cluster into the Pipeline
- Kafka cluster can be remote or local
- See :py:obj:`~morpheus.pipeline.input.from_kafka.KafkaSourceStage` for more information
- Refer to :py:obj:`~morpheus.pipeline.input.from_kafka.KafkaSourceStage` for more information
* ``from-file``
- Reads from a local file into the Pipeline
- Supports JSON lines format
- All lines are read at the start and queued into the pipeline at one time. Useful for performance testing.
- See :py:obj:`~morpheus.pipeline.input.from_file.FileSourceStage` for more information
- Refer to :py:obj:`~morpheus.pipeline.input.from_file.FileSourceStage` for more information

Stages
^^^^^^
Expand Down Expand Up @@ -74,7 +74,7 @@ After the ``====Building Pipeline====`` message, if logging is ``INFO`` or great
stages and the type transformations of each stage. To be a valid Pipeline, the output type of one stage must match the
input type of the next. Many stages are flexible and will determine their type at runtime but some stages require a
specific input type. If your Pipeline is configured incorrectly, Morpheus will report the error. For example, if we run
the same command as above but forget the ``serialize`` stage, you will see the following:
the same command as above but forget the ``serialize`` stage, the following will be displayed:

.. code-block:: console
Expand All @@ -94,7 +94,7 @@ the same command as above but forget the ``serialize`` stage, you will see the f
This indicates that the ``to-file`` stage cannot accept the input type of `morpheus.pipeline.messages.MultiMessage`.
This is because the ``to-file`` stage has no idea how to write that class to a file, it only knows how to write strings.
To ensure you have a valid pipeline, look at the ``Accepted input types: (typing.List[str],)`` portion of the message.
To ensure you have a valid pipeline, examine the ``Accepted input types: (typing.List[str],)`` portion of the message.
This indicates you need a stage that converts from the output type of the ``deserialize`` stage,
`morpheus.pipeline.messages.MultiMessage`, to `typing.List[str]`, which is exactly what the ``serialize`` stage does.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ def setup(app):

# The following is used by sphinx.ext.linkcode to provide links to github
linkcode_resolve = make_linkcode_resolve(
'morpheus', 'https://github.com/NVIDIA/Morpheus'
'morpheus', 'https://github.com/nv-morpheus/Morpheus'
'morpheus/-/blob/{revision}/'
'{package}/{path}#L{lineno}')

Expand Down
1 change: 1 addition & 0 deletions docs/source/developer_guide/examples/abp_pcap_detection.md
15 changes: 15 additions & 0 deletions docs/source/developer_guide/examples/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
========
Examples
========

.. toctree::
:maxdepth: 20

abp_nvsmi_detection.md
abp_pcap_detection.md
gnn_fraud_detection_pipeline.md
log_parsing.md
nlp_si_detection.md
ransomware_detection.md
root_cause_analysis.md
sid_visualization.md
1 change: 1 addition & 0 deletions docs/source/developer_guide/examples/log_parsing.md
1 change: 1 addition & 0 deletions docs/source/developer_guide/examples/nlp_si_detection.md
1 change: 1 addition & 0 deletions docs/source/developer_guide/examples/sid_visualization.md
10 changes: 5 additions & 5 deletions docs/source/developer_guide/guides/1_simple_python_stage.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ To start, we will implement a single stage that could be included in a pipeline.

Defining this stage requires us to specify the stage type. Morpheus stages contain a single input and a single output inherited from `SinglePortStage`. Stages that act as sources of data, in that they do not take an input from a prior stage but rather produce data from a source such as a file, Kafka service, or other external sources, will need to inherit from the `SingleOutputSource` base class.

Optionally, stages can be registered as a command with the Morpheus CLI using the `register_stage` decorator. This allows for pipelines to be constructed from both pre-built stages and custom user stages via the command line. Any constructor arguments will be introspected using [numpydoc](https://numpydoc.readthedocs.io/en/latest/) and exposed as command line flags. Similarly the class's docstrings will be exposed in the help string of the stage on the command line.
Optionally, stages can be registered as a command with the Morpheus CLI using the `register_stage` decorator. This allows for pipelines to be constructed from both pre-built stages and custom user stages via the command line. Any constructor arguments will be introspected using [numpydoc](https://numpydoc.readthedocs.io/en/latest/) and exposed as command line flags. Similarly the class's docstrings will be exposed in the help string of the stage on the command line.

We start our class definition with a few basic imports:

Expand All @@ -54,7 +54,7 @@ There are four methods that need to be defined in our new subclass to implement
return "pass-thru"
```

The `accepted_types` method returns a tuple of message classes that this stage accepts as valid inputs. Morpheus uses this to validate that the parent of this stage emits a message that this stage can accept. Since our stage is a pass through, we will declare that we can accept any incoming message type. Note that production stages will often declare only a single Morpheus message class such as `MessageMeta` or `MultiMessage` (see the message classes defined in `morpheus.pipeline.messages` for a complete list).
The `accepted_types` method returns a tuple of message classes that this stage accepts as valid inputs. Morpheus uses this to validate that the parent of this stage emits a message that this stage can accept. Since our stage is a pass through, we will declare that we can accept any incoming message type. Note that production stages will often declare only a single Morpheus message class such as `MessageMeta` or `MultiMessage` (refer to the message classes defined in `morpheus.pipeline.messages` for a complete list).
```python
def accepted_types(self) -> typing.Tuple:
return (typing.Any,)
Expand Down Expand Up @@ -113,7 +113,7 @@ class PassThruStage(SinglePortStage):
"""
A Simple Pass Through Stage
"""

@property
def name(self) -> str:
return "pass-thru"
Expand Down Expand Up @@ -182,7 +182,7 @@ Finally, we run the pipeline:
pipeline.run()
```

The output should look like this:
The output should display:
```
====Registering Pipeline====
====Registering Pipeline Complete!====
Expand Down Expand Up @@ -255,4 +255,4 @@ morpheus --log_level=debug --plugin examples/developer_guide/1_simple_python_sta
from-file --filename=examples/data/email_with_addresses.jsonlines \
pass-thru \
monitor
```
```
12 changes: 6 additions & 6 deletions docs/source/developer_guide/guides/2_real_world_phishing.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ limitations under the License.

## Data Preprocessing

Now that we've seen a basic example of how to create a stage and use it in the context of a pipeline, we'll move on to a more advanced example that is representative of what we might want to do in a real-world situation. Given a set of records, each of which represents an email, suppose we want to predict which records correspond to fraudulent emails.
The previous example demonstated how to create a simple stage and use it in the context of a pipeline, we'll move on to a more advanced example that is representative of what we might want to do in a real-world situation. Given a set of records, each of which represents an email, suppose we want to predict which records correspond to fraudulent emails.

As part of this process, we might want to use a classification model trained on various pieces of metadata, such as recipient count, in addition to the raw content of each email. If we suppose this is true for our example, we need to build and connect a pre-processing stage to attach this information to each record before applying our classifier.

Expand All @@ -28,7 +28,7 @@ For this task, we'll need to define a new stage, which we will call our `Recipie
1. Count the number of recipients in the email's metadata.
1. Construct a Morpheus `MessageMeta` object that will contain the record content along with the augmented metadata.

For this stage, the code will look very similar to the previous example with a few notable changes. We will be working with the `MessageMeta` class. This is a Morpheus message containing a [cuDF](https://docs.rapids.ai/api/cudf/stable/) [DataFrame](https://docs.rapids.ai/api/cudf/stable/api_docs/dataframe.html). Since we will expect our new stage to operate on {py:obj}`~morpheus.pipeline.messages.MessageMeta` types, our new `accepted_types` method now looks like:
For this stage, the code will be similar to the previous example with a few notable changes. We will be working with the `MessageMeta` class. This is a Morpheus message containing a [cuDF](https://docs.rapids.ai/api/cudf/stable/) [DataFrame](https://docs.rapids.ai/api/cudf/stable/api_docs/dataframe.html). Since we will expect our new stage to operate on {py:obj}`~morpheus.pipeline.messages.MessageMeta` types, our new `accepted_types` method now looks like:

```python
def accepted_types(self) -> typing.Tuple:
Expand Down Expand Up @@ -59,7 +59,7 @@ def on_data(self, message: MessageMeta) -> MessageMeta:
return message
```

If mutating the data frame is undesirable, we could make a call to the data frame's [copy](https://docs.rapids.ai/api/cudf/stable/api_docs/api/cudf.DataFrame.copy.html#cudf.DataFrame.copy) method and return a new `MessageMeta`. Note however that this would come at the cost of performance and increased memory usage. Our updated `on_data` method would look like this (changing the first and last lines of the method):
If mutating the data frame is undesirable, we could make a call to the data frame's [copy](https://docs.rapids.ai/api/cudf/stable/api_docs/api/cudf.DataFrame.copy.html#cudf.DataFrame.copy) method and return a new `MessageMeta`. Note however that this would come at the cost of performance and increased memory usage. Changing the first and last lines of the `on_data` method to:

```python
def on_data(self, message: MessageMeta) -> MessageMeta:
Expand Down Expand Up @@ -175,7 +175,7 @@ Output:
{"name":"phishing-bert-onnx","versions":["1"],"platform":"onnxruntime_onnx","inputs":[{"name":"input_ids","datatype":"INT64","shape":[-1,128]},{"name":"attention_mask","datatype":"INT64","shape":[-1,128]}],"outputs":[{"name":"output","datatype":"FP32","shape":[-1,2]}]}
```

From this information, we can see that the expected shape of the model inputs is `"shape":[-1,128]}`.
From this information, we note that the expected shape of the model inputs is `"shape":[-1,128]}`.

### Defining our Pipeline
Let's set up the paths for our input and output files. For simplicity, we assume that the `MORPHEUS_ROOT` environment variable is set to the root of the Morpheus project repository. In a production deployment, it may be more prudent to replace our usage of environment variables with command-line flags or a dedicated configuration management library.
Expand Down Expand Up @@ -596,7 +596,7 @@ In Morpheus, we define a stage to be a sink if it outputs the results of a pipel

Recall that in the previous section we wrote a `RabbitMQSourceStage`. We will now complement that by writing a sink stage that can output Morpheus data into RabbitMQ. For this example, we are again using the [pika](https://pika.readthedocs.io/en/stable/#) client for Python.

The code for our sink will look similar to other stages with a few changes. First, we will subclass `SinglePortStage`:
The code for our sink will be similar to other stages with a few changes. First, we will subclass `SinglePortStage`:

```python
@register_stage("to-rabbitmq")
Expand Down Expand Up @@ -719,4 +719,4 @@ class WriteToRabbitMQStage(SinglePortStage):
```

## Note
For information about testing the `RabbitMQSourceStage` and `WriteToRabbitMQStage` stages see [examples/developer_guide/2_2_rabbitmq/README.md](../../../../examples/developer_guide/2_2_rabbitmq/README.md)
For information about testing the `RabbitMQSourceStage` and `WriteToRabbitMQStage` stages refer to [examples/developer_guide/2_2_rabbitmq/README.md](../../../../examples/developer_guide/2_2_rabbitmq/README.md)
6 changes: 3 additions & 3 deletions docs/source/developer_guide/guides/3_simple_cpp_stage.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ To start with, we have our Morpheus and SRF-specific includes:
#include <srf/segment/object.hpp> // for Segment Object
```

We'll want to define our stage in its own namespace. In this case, we will name it `morpheus_example`, giving us a namespace and class definition that look like this:
We'll want to define our stage in its own namespace. In this case, we will name it `morpheus_example`, giving us a namespace and class definition like:

```cpp
namespace morpheus_example {
Expand Down Expand Up @@ -192,7 +192,7 @@ The constructor for our class is responsible for passing the output of `build_op
PassThruStage::PassThruStage() : PythonNode(base_t::op_factory_from_sub_fn(build_operator())) {}
```
We can see that the output of `build_operator()` is not passed directly to the `PythonNode` constructor and instead gets passed to `base_t::op_factory_from_sub_fn()`. This is because reactive operators can be defined two ways:
Note that the output of `build_operator()` is not passed directly to the `PythonNode` constructor and instead gets passed to `base_t::op_factory_from_sub_fn()`. This is because reactive operators can be defined two ways:
1. Using the short form `std::function<rxcpp::observable<T>(rxcpp::observable<R>)` which is good when you can use an existing `rxcpp` operator
2. Using the long form `std::function<rxcpp::subscription(rxcpp::observable<T>, rxcpp::subscriber<R>)>` which allows for more customization and better control over the lifetime of objects.
Expand Down Expand Up @@ -253,7 +253,7 @@ std::shared_ptr<srf::segment::Object<PassThruStage>> PassThruStageInterfaceProxy
}
```
The Python interface itself defines a Python module named `morpheus_example` and a Python class in that module named `PassThruStage`. Note that the only method we are exposing to Python is the constructor. Our Python code will see this class as `lib_.morpheus_example.PassThruStage`.
The Python interface itself defines a Python module named `morpheus_example` and a Python class in that module named `PassThruStage`. Note that the only method we are exposing to Python is the constructor. The class will be exposed to Python code as `lib_.morpheus_example.PassThruStage`.
```cpp
namespace py = pybind11;
Expand Down
4 changes: 2 additions & 2 deletions docs/source/developer_guide/guides/4_source_cpp_stage.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ std::string m_queue_name;
AmqpClient::Channel::ptr_t m_channel;
```

Wrapping it all together, our header file should look like this:
Wrapping it all together, our header file should be similar to:
```cpp
#pragma once

Expand Down Expand Up @@ -335,7 +335,7 @@ PYBIND11_MODULE(morpheus_rabbit, m)
}
```
Wrapping it all together, our source file should look like:
Wrapping it all together, our source file should be:
```cpp
#include "rabbitmq_source.hpp"
Expand Down
Loading

0 comments on commit 1fec0a5

Please sign in to comment.