Morpheous Data Generation

[shubh0155]

Hello I have installed Morpheous using docker and it is running succesfully on my centos os system. but i am unable to generate the data set. How to generate the data set from nvidia-smi command as the data given in examples has 176 columns ? is there any way apart from NETQ agent to generate that data set.

[jarmak-nv]

Hi @shubh0155!

Thanks for reaching out to us - just a few more details to help us dig in please:

• Can you share exactly which example you're trying to run and which dataset you're referencing with links ideally
• Can you share examples of what you have tried
• Can you share any log output that could help us figure out how to help

[shubh0155]

Hello @jarmak-nv, thank you for your quick response.

I am currently working on implementing Anomalous Behavior Profiling using Morpheus and need assistance. The GitHub link I am referring to is: https://github.com/nv-morpheus/Morpheus/tree/branch-23.11/examples/abp_nvsmi_detection

The issue I am facing is that the nvidia-smi command is not generating the required 176 columns of data needed for detecting Anomalous Behavior. The necessary dataset can be found here: https://github.com/nv-morpheus/Morpheus/tree/branch-23.11/examples/data (filename: nvsmi.jsonfiles).

However, the data generated by the nvidia-smi command on my CentOS OS system consists of only 6-7 columns.

I would appreciate your help in identifying how I can generate the complete dataset with 176 columns on my CentOS OS system.

Thank you in advance for your assistance.

[ABHIPATEL98]

Hello @jarmak-nv,

Both @shubh0155 and I are working on a project together. To provide you with a clear understanding, our system is equipped with an Nvidia P5000 GPU. We have successfully installed Nvidia Morpheus using a Docker container and tested it with a sample example called "abp_nvsmi_detection" using the provided sample dataset named "nvsmi.jsonlines."

Now, we want to use our own data to test the "abp_nvsmi_detection" model. However, we encountered an issue when trying to generate the dataset similar to "nvsmi.jsonlines." The problem is that the "nvsmi.jsonlines" file contains around 176 parameters that we are unable to generate using the "nvidia-smi" or "nvidia-smi dmon" commands.

Therefore, we kindly request your assistance in guiding us on the available methods to generate a dataset like "nvsmi.jsonlines" so that we can effectively test our system with the "abp_nvsmi_detection" model.
which version of driver we are using I am attaching a screenshot of output generated by nvidia-smi.

We would appreciate it if you could help us with this task.

[mdemoret-nv]

@shubh0155 and @ABHIPATEL98 The data for that example does not come from nvidia-smi but instead comes from a NetQ agent designed to monitor entire clusters of GPUs. We should have made this more clear in the documentation. The reference to nvidia-smi was just to give the user some background information on what type of statistics were included in the sample dataset.

However, all of the necessary information can be pulled from nvidia-smi and written to a file in the correct format. Let me see if I can try to make a script that will work for this example.

[mdemoret-nv]

@shubh0155 and @ABHIPATEL98 Can you try the following script to generate a dataset from your local GPUs? It uses nvidia-ml which is the same library used by nvidia-smi. You will need to install nvidia-ml-py3 and pandas for this script to work.

import time

import pandas as pd
from pynvml.smi import NVSMI_QUERY_GPU
from pynvml.smi import nvidia_smi

# Output name
output_file = "nvsmi.json"

# Interval
interval_ms = 1000

query_opts = NVSMI_QUERY_GPU.copy()

# Remove the timestamp and supported clocks from the query
del query_opts["timestamp"]
del query_opts["supported-clocks"]

nvsmi = nvidia_smi.getInstance()

with open(output_file, "w", encoding="UTF-8") as f:

    while (True):

        dq = nvsmi.DeviceQuery(list(query_opts.values()))

        output_dicts = []

        # Flatten the GPUs to allow for a new row per GPU
        for gpu in dq["gpu"]:
            single_gpu = dq.copy()

            # overwrite the gpu list with a single gpu
            single_gpu["gpu"] = gpu

            output_dicts.append(single_gpu)

        df = pd.json_normalize(output_dicts, record_prefix="nvidia_smi_log")

        # Rename the id column to match the XML converted output from NetQ
        df.rename(columns={"gpu.id": "gpu.@id", "count": "attached_gpus"}, inplace=True)

        df.rename(columns=lambda x: "nvidia_smi_log" + "." + x, inplace=True)

        # Add the current timestamp
        df.insert(0, "timestamp", time.time())

        df.to_json(f, orient="records", lines=True)

        f.flush()

        time.sleep(interval_ms / 1000.0)

This will write a new entry to the output file once per second until you press Ctrl+C to exit.

[mdemoret-nv]

I've also added an issue to include this in the example documentation: #1097

[shubh0155]

Dear @mdemoret-nv

Thank you for your valuable assistance and guidance with the script to generate a dataset from local GPUs. Your expertise and prompt response were truly appreciated. Your script worked perfectly, to give us an idea about the required data set and we have requested for the NETQ access to proceed ahead . Your contributions to the community are commendable, and I'm grateful for your support.

[ABHIPATEL98]

Dear @mdemoret-nv Thank you for your response. We have run the script you provided and it is working fine. However, most of the values are coming back as N/A. When we give the generated data from the script to the "abp_nvsmi_detection" model, it produces an error: "Inference Rate[Complete]: 0 inf [00:00, ? inf/s]". The error message indicates that the model is missing the "nvidia_smi_log.gpu.pci.tx_util" column. Since NetQ is not open source, we are unable to generate new sample data for the ABP nvsmi detection example. Is there any other alternative way to do this?
Thank you for your help in advance.