Provide migration path for monitoring alerting from Watcher to Kibana alerting

[jakelandis]

A new alerting system in Kibana is currently under development : https://www.elastic.co/blog/alerting-in-the-elastic-stack

As part of the this effort we want to enable users a migration path for the default Watches that ship with Monitoring to this new system. This requires some way for Kibana to tell Elasticsearch it is now handling the default alerts. Once Kibana is handling the default alerts, Elasticsearch should a) remove existing default monitoring Watches b) prevent new default monitoring Watches from being used.

Note - this only includes the 6 default Watches that ship with monitoring and does not include custom watches.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Monitoring)

[chrisronline]

Any update on when this might be available? Does 7.8 look realistic?

[chrisronline]

To provide a little color to this issue, I wanted to explain the need for this for all interested parties.

Elasticsearch exporters will attempt to create the six monitoring cluster alert watches every time the exporter is executed (on a master node). This code will check a few notable things before creating the watches:

1. Does the user have the right license for watcher? Source
2. Is the watch in the blacklist? If it is, skip adding it. Source

The blacklisting feature is configured like:

cluster_alerts.management.blacklist
Prevents the creation of specific cluster alerts. It also removes any applicable watches that already exist in the current cluster.
You can add any of the following watch identifiers to the blacklist:

elasticsearch_cluster_status
elasticsearch_version_mismatch
elasticsearch_nodes
kibana_version_mismatch
logstash_version_mismatch
xpack_license_expiration
For example: ["elasticsearch_version_mismatch","xpack_license_expiration"].

This feature is nice, but it's configured at the exporter level (instead of at the Elasticsearch cluster level) meaning that the blacklist only applies for existing exporters. If a user were to add a new exporter in the future, this blacklist wouldn't copy over.

So we aren't left with many options to disable these alerts. We can disable watcher on the cluster, but that will also disable any custom watches the user has setup.

We need a way to delete any existing watcher-based cluster alerts, as well as a way to disable new ones from being created.

FYI, there is some background in this thread.

Please correct me if I'm mistaken on any of these points, as I'm not an expert with how Elasticsearch does this.

[ravikesarwani]

Had a quick sync up to discuss this issue between @jakelandis @chrisronline @igoristic @ravikesarwani.
We will try to get this code delivered for 7.10 with both changes on ES and SM.

• ES team will provide an API/cluster setting that when set will remove the 6 watchers and also modify the code so that the 6 watcher's aren't created again.
• Stack monitoring team will create the 6 replacement Kibana alerts when user visits the stack monitoring UI and will call the API/cluster setting that ES team provides.

[ravikesarwani]

@jakelandis Any update on the progress on the API? Is this getting completed for 7.10?

[sgrodzicki]

@jakelandis Any update on the progress on the API? Is this getting completed for 7.10?

Also, is this the right issue to track or will there be another implementation issue?