Version

streaming ransomare model

Which installation method(s) does this occur on?

No response

Describe the bug.

We have a service in the DPU that is extracting raw data (memory features snapshots) and transmit it to kafka - all the time. The time between 2 memory snapshot is ~5 seconds. So, the ransomware detection pipeline should preprocess and inference each snapshot with less than 5 seconds, for not suffering from exploding delay. To test it, if we are suffering from an exploding delay, I monitor the kafka input to watch the input snapshot ID and in the same time I monitor the kafka output to watch the output snapshot ID.

It seems like, we are suffering from a weird phenomenon that the difference between the IDs is increasing but we also are receiving pulses of huge batches of messages to kafka output that is decreasing the difference between the IDs, so even after a long time, the difference is not exploding, but we are suffering from huge delays between the current input snapshot and the output snapshot. In addition, we have a maximum for this delay - 50 snapshots (4-3 minutes)… In all the experiment that I did, we never crossed the ~50 snapshot delay…

An example to explain my description:
In the beginning, the input snapshot ID is 1 and the output snapshot ID is 1.
After 3 minutes, input snapshot is 36 and output snapshot is 10.
After another 1 minute, input snapshot is 48 and output snapshot is 40.
After another 2 minutes, input snapshot is 72 and output snapshot is 46.
After another 2 minutes, input snapshot is 96 and output snapshot is 90.
And so on…
@bsuryadevara

Minimum reproducible example

No response

Relevant log output

No response

Full env printout

No response

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report