initial commit for remote monitor support #1547

sbcd90 · 2024-05-15T19:41:23Z

Issue #, if available:
#1546

Description of changes:

CheckList:

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep · 2024-05-16T18:55:08Z

...e-remote-monitor-plugin/src/main/java/org/opensearch/alerting/SampleRemoteMonitorPlugin.java

+    }
+
+    @Override
+    public String getMonitorType() {


plugins should be allowed to add multiple types of monitors.
SAP itself probably will implement

threat intel ioc scan monitor

ueba monitor

rules monitor

bucket level monitor with findings and pipeline aggs

added support for it. https://github.com/opensearch-project/alerting/pull/1547/files#diff-e69834c03ec992bfc16754998279e319b8bba7a80ef208a6666b8066c283cf55R79

eirsep

Can we plz add the skeleton for a fan-out and monitor metadata capabilities of doc level monitor
also how to define new triggers? triggers seem tightly coupled with the existing defintions.

eirsep · 2024-05-16T19:07:07Z

alerting/build.gradle

@@ -3,6 +3,7 @@
 * SPDX-License-Identifier: Apache-2.0
 */

+import com.github.jengelman.gradle.plugins.shadow.ShadowPlugin


is this the same dependency used by jobscheduler or notification for shadow plugin?

eirsep · 2024-05-22T08:43:37Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt

@@ -185,10 +187,16 @@ object MonitorMetadataService :

    suspend fun recreateRunContext(metadata: MonitorMetadata, monitor: Monitor): MonitorMetadata {
        try {
-            val monitorIndex = if (monitor.monitorType == Monitor.MonitorType.DOC_LEVEL_MONITOR)
+            val monitorIndex = if (
+                monitor.isMonitorOfStandardType() &&


why not just check string value is doc level monitor?

just want to use a single string constant, so just reusing the enum value.

eirsep · 2024-05-22T08:43:59Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt

            (monitor.inputs[0] as DocLevelMonitorInput).indices[0]
        else null
-        val runContext = if (monitor.monitorType == Monitor.MonitorType.DOC_LEVEL_MONITOR && createWithRunContext)
+        val runContext = if (
+            monitor.isMonitorOfStandardType() &&


extract into method - isDocLevelMonitor()

simplified this logic https://github.com/opensearch-project/alerting/pull/1547/files#diff-5d88a62bc4191f374a4813c2256646402800ae1beed484e7a1812b7445957308R191

eirsep · 2024-05-22T08:46:55Z

alerting/src/main/kotlin/org/opensearch/alerting/remote/monitors/RemoteMonitorRegistry.kt

+
+import org.opensearch.alerting.spi.RemoteMonitorRunner
+
+class RemoteMonitorRegistry(val monitorType: String, val monitorRunner: RemoteMonitorRunner)


ok updated it.

sbcd90 · 2024-05-28T02:20:01Z

Can we plz add the skeleton for a fan-out and monitor metadata capabilities of doc level monitor also how to define new triggers? triggers seem tightly coupled with the existing defintions.

added in latest changes.

Monitor metadata support
https://github.com/opensearch-project/alerting/blob/0181b855a8c1e4badc8f2d00cad17fc068bcf879/alerting/src/main/kotlin/org/opensearch/alerting/remote/monitors/RemoteDocumentLevelMonitorRunner.kt
Fan-out support

alerting/spi/src/main/kotlin/org/opensearch/alerting/spi/RemoteMonitorRunner.kt

Line 45 in 0181b85

open suspend fun doFanOut(

Now, Remote Doc-Level monitors just need to implement `fan out transport actions.
https://github.com/opensearch-project/alerting/blob/0181b855a8c1e4badc8f2d00cad17fc068bcf879/sample-remote-monitor-plugin/src/main/java/org/opensearch/alerting/monitor/runners/SampleRemoteDocLevelMonitorRunner.java
https://github.com/opensearch-project/alerting/blob/0181b855a8c1e4badc8f2d00cad17fc068bcf879/sample-remote-monitor-plugin/src/main/java/org/opensearch/alerting/monitor/fanouts/TransportRemoteDocLevelMonitorFanOutAction.java

Remote Doc-level monitors can be created in same way like regular doc-level monitors.
https://github.com/opensearch-project/alerting/blob/0181b855a8c1e4badc8f2d00cad17fc068bcf879/sample-remote-monitor-plugin/src/main/java/org/opensearch/alerting/SampleRemoteMonitorRestHandler.java#L178-%23L203

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep · 2024-05-28T20:17:13Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt

@@ -185,10 +185,10 @@ object MonitorMetadataService :

    suspend fun recreateRunContext(metadata: MonitorMetadata, monitor: Monitor): MonitorMetadata {
        try {
-            val monitorIndex = if (monitor.monitorType == Monitor.MonitorType.DOC_LEVEL_MONITOR)
+            val monitorIndex = if (monitor.monitorType.endsWith(Monitor.MonitorType.DOC_LEVEL_MONITOR.value))


can we just have a separate flag instead of depending on name

check can be monitor.monitorType == Monitor.MonitorType.DOC_LEVEL_MONITOR || "use_doc_level_feature_bool" == true ?

eirsep · 2024-05-28T21:29:30Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexWorkflowAction.kt

@@ -400,7 +402,9 @@ class TransportIndexWorkflowAction @Inject constructor(
                        log.warn("Metadata doc id:${monitorMetadata.id} exists, but it shouldn't!")
                    }

-                    if (monitor.monitorType == Monitor.MonitorType.DOC_LEVEL_MONITOR) {
+                    if (


can workflow add any kind of remote monitor?

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep · 2024-05-29T02:04:09Z

how will index validate mapping of the custom trigger and custom input

eirsep · 2024-05-29T02:10:25Z

how will serialization work for custom trigger and custom input

https://github.com/opensearch-project/common-utils/blob/main/src/main/kotlin/org/opensearch/commons/alerting/model/Trigger.kt#L50-L63

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 · 2024-05-30T02:45:31Z

how will serialization work for custom trigger and custom input

https://github.com/opensearch-project/common-utils/blob/main/src/main/kotlin/org/opensearch/commons/alerting/model/Trigger.kt#L50-L63

here is an example how to define our own inputs & triggers in remote monitors.
custom input - https://github.com/opensearch-project/alerting/pull/1547/files#diff-80a0de70d190d8c5ba474303bbbdc9782a65c6173e3f6722bc3fb350c745bc83
custom trigger - https://github.com/opensearch-project/alerting/pull/1547/files#diff-04bd41521bcc623cf8018e11ed9ceefb09e59a76ce204a191795e85d4ccabca1
defining a monitor using them - https://github.com/opensearch-project/alerting/blob/629a0c30a1a867129df99b1fb4821cdc4fdcfd0d/sample-remote-monitor-plugin/src/main/java/org/opensearch/alerting/SampleRemoteMonitorRestHandler.java#L67-%23L96
accessing them in custom monitor runner -
custom input - https://github.com/opensearch-project/alerting/blob/629a0c30a1a867129df99b1fb4821cdc4fdcfd0d/sample-remote-monitor-plugin/src/main/java/org/opensearch/alerting/monitor/runners/SampleRemoteMonitorRunner1.java#L69-%23L75
custom trigger - https://github.com/opensearch-project/alerting/blob/629a0c30a1a867129df99b1fb4821cdc4fdcfd0d/sample-remote-monitor-plugin/src/main/java/org/opensearch/alerting/monitor/runners/SampleRemoteMonitorRunner1.java#L77-%23L83

there are more complex examples in the sample plugin e.g. using nested objects.

eirsep · 2024-05-30T21:28:07Z

alerting/src/main/kotlin/org/opensearch/alerting/AlertingPlugin.kt

@@ -240,6 +241,7 @@ internal class AlertingPlugin : PainlessExtension, ActionPlugin, ScriptPlugin, R
            ClusterMetricsInput.XCONTENT_REGISTRY,
            DocumentLevelTrigger.XCONTENT_REGISTRY,
            ChainedAlertTrigger.XCONTENT_REGISTRY,
+            RemoteMonitorTrigger.XCONTENT_REGISTRY,


do we also need RemoteMonitorInput xcontent registry?

if yes, can you add appropriate tests to serialize

we dont need RemoteMonitorInput because in common-utils Input.parse doesn't use it. https://github.com/opensearch-project/common-utils/blob/main/src/main/kotlin/org/opensearch/commons/alerting/model/Input.kt#L30
however, in Trigger.parse it is used. https://github.com/opensearch-project/common-utils/blob/main/src/main/kotlin/org/opensearch/commons/alerting/model/Trigger.kt#L41

eirsep

can you check xcontent registry

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 · 2024-05-31T07:49:27Z

can you check xcontent registry

responded here. #1547 (comment)

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 · 2024-05-31T21:24:06Z

all builds pass. merging pr.

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

initial commit for remote monitor support

197b7f8

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 requested review from lezzago, AWSHurneyt, eirsep, getsaurabh02, praveensameneni, qreshi, bowenlan-amzn, rishabhmaurya, engechas, riysaxen-amzn and jowg-amazon as code owners May 15, 2024 19:41

sbcd90 marked this pull request as draft May 15, 2024 19:41

eirsep reviewed May 16, 2024

View reviewed changes

eirsep requested changes May 16, 2024

View reviewed changes

sbcd90 marked this pull request as ready for review May 22, 2024 03:47

sbcd90 force-pushed the remote_monitor branch 2 times, most recently from 8317f29 to abcc7ce Compare May 22, 2024 03:57

eirsep reviewed May 22, 2024

View reviewed changes

changes to add support for remote monitors in alerting

0181b85

sbcd90 force-pushed the remote_monitor branch from ad116fb to 0181b85 Compare May 25, 2024 03:39

sbcd90 marked this pull request as draft May 25, 2024 03:41

changes to cleanup the pr

70f165f

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 marked this pull request as ready for review May 28, 2024 18:50

fix ktlint for windows

8bd00b4

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep reviewed May 28, 2024

View reviewed changes

sbcd90 added 2 commits May 29, 2024 00:41

update alerting with latest common-utils commit

a840958

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

update test-workflow

315d8ac

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

changes to support generic inputs and triggers in remote monitors

629a0c3

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep reviewed May 30, 2024

View reviewed changes

eirsep requested changes May 30, 2024

View reviewed changes

attempt to fix windows builds

f48253c

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

eirsep approved these changes May 31, 2024

View reviewed changes

AWSHurneyt approved these changes May 31, 2024

View reviewed changes

sbcd90 added 5 commits May 31, 2024 19:59

fix windows builds

77a83ff

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Merge branch 'main' into remote_monitor

4b88443

fix merge conflicts

ae13a15

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

fix ktlint issues

5564e3a

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

fix windows build

58a6f29

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

sbcd90 merged commit 5ed4db9 into opensearch-project:main May 31, 2024
18 checks passed

AWSHurneyt pushed a commit to AWSHurneyt/OpenSearch-Alerting that referenced this pull request Jun 25, 2024

initial commit for remote monitor support (opensearch-project#1547)

fc2b8ec

Signed-off-by: Subhobrata Dey <sbcd90@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial commit for remote monitor support #1547

initial commit for remote monitor support #1547

sbcd90 commented May 15, 2024 •

edited

Loading

eirsep May 16, 2024 •

edited

Loading

sbcd90 May 28, 2024

eirsep left a comment

eirsep May 16, 2024

sbcd90 May 28, 2024

eirsep May 22, 2024

sbcd90 May 28, 2024

eirsep May 22, 2024 •

edited

Loading

sbcd90 May 28, 2024

eirsep May 22, 2024

sbcd90 May 28, 2024

sbcd90 commented May 28, 2024 •

edited

Loading

eirsep May 28, 2024

eirsep May 28, 2024

eirsep commented May 29, 2024

eirsep commented May 29, 2024

sbcd90 commented May 30, 2024

eirsep May 30, 2024

sbcd90 May 31, 2024

eirsep left a comment

sbcd90 commented May 31, 2024

sbcd90 commented May 31, 2024


		import org.opensearch.alerting.spi.RemoteMonitorRunner

		class RemoteMonitorRegistry(val monitorType: String, val monitorRunner: RemoteMonitorRunner)

initial commit for remote monitor support #1547

initial commit for remote monitor support #1547

Conversation

sbcd90 commented May 15, 2024 • edited Loading

eirsep May 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep May 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbcd90 commented May 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep commented May 29, 2024

eirsep commented May 29, 2024

sbcd90 commented May 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eirsep left a comment

Choose a reason for hiding this comment

sbcd90 commented May 31, 2024

sbcd90 commented May 31, 2024

sbcd90 commented May 15, 2024 •

edited

Loading

eirsep May 16, 2024 •

edited

Loading

eirsep May 22, 2024 •

edited

Loading

sbcd90 commented May 28, 2024 •

edited

Loading