[FEATURE] Remote Monitors in OpenSearch Alerting

[sbcd90]

Introduction

OpenSearch Alerting plugin allows users to monitor different data sources (e.g. indices) & subsequently create alerts when the trigger condition is met. OpenSearch Alerting plugin today supports 4 default monitor types.

Query Level Monitors

The Query Level Monitors in OpenSearch Alerting allows user to periodically run a OpenSearch search query against an index storing log documents. Alerts are triggered when a particular trigger condition is met.

Bucket Level Monitors

The Bucket Level Monitors are used when aggregation queries are periodically run to monitor an index(or set of indices) containing log documents. Findings & alerts are triggered when a particular trigger condition on the aggregated buckets are met.

Document Level Monitors

Document level monitors can be briefly defined in 2 steps. In the first step, documents created within a fixed time interval in a customer created index are matched against a set of queries. Now, in the second step, these document-query pairs are matched against a trigger condition & if it is match, an alert is triggered.

Cluster Metric Monitors

Per cluster metrics monitors are a type of alert monitor that collects and analyzes metrics from a single cluster, providing insights into the cluster’s performance and health.

Apart from these 4 predefined monitor types which provides specific functionalities, the OpenSearch Alerting plugin infrastructure can be re-used to run ad-hoc tasks periodically.

This RFC addresses this issue of how OpenSearch Alerting plugin infrastructure can be leveraged by downstream OpenSearch plugins to model their monitoring use-cases.

Design

The OpenSearch Alerting plugin is primarily composed of 2 distinct modules.

• Alerting module - This module handles the rest handler & transport handler layers as well as handles the core logic of managing the monitor runners.
• Core module - This module handles the job-scheduler layer to periodically trigger & schedule jobs(or monitor runs) in different OS nodes.

In addition to these 2 core modules , we introduce a new module named alerting-spi which exposes a set of interfaces which downstream OpenSearch plugins can leverage to integrate with OpenSearch Alerting plugin infrastructure.

This avoids downstream plugins to add heavy core alerting modules as direct dependency.
Here is a high-level component diagram of how a downstream OS plugin can integrate & leverage OpenSearch Alerting plugin infrastructure to meet specific monitoring needs.