[Monitoring] Migrate cluster alerts from watcher to Kibana alerting

[chrisronline]

Depends on elastic/elasticsearch#50032

Action items

• Update Kibana to call this new API [Monitoring] Remove deprecated watcher-based cluster alerts #85047
• Update existing legacy cluster alerts to fetch data from .monitoring-{stackProduct}-* documents instead of from .monitoring-alerts* documents [Monitoring] Migrate data source for legacy alerts to monitoring data directly #87377
• Communicate change to user and tell them to may need to add connectors (see [Monitoring] Migrate cluster alerts from watcher to Kibana alerting #81020 (comment)) [Monitoring] Add toast for newly created alerts #89202

Background

We need to leverage the new ES api to remove all existing cluster alert watches and let our new Kibana alerts work by themselves.

Right now, we fetch legacy alert data from the result of the cluster alert watches index (which is .monitoring-alerts-*). Once the watches are gone, these alerts need to fetch the data manually, but I have done this work already:

• Elasticsearch version mismatch alert
• Kibana version mismatch alert
• Logstash version mismatch alert
• Cluster health alert
• Nodes changed alert
• License expiration alert

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[jakelandis]

elastic/elasticsearch#50032 is almost complete..but not quite ready yet. However, the API should be pretty well sorted if you want to start to code against that.

The request will be POST /_monitoring/migrate/alerts and the response will look like this:

{
  "exporters":[
    {
      "name":"thename",
      "type":"http|local",
      "migration_complete":true,
      "reason":"optional - exception"
    },
    {
      "name":"thename2",
      "type":"http|local",
      "migration_complete":false,
      "reason":"optional - exception"
    }
  ]
}

[chrisronline]

@ravikesarwani

Should we wait on #85047 and #87377 until the Elasticsearch team implements changes to detect if users have actions connected to watcher (aka: elastic/elasticsearch#50032 (comment))? If we don't wait, it means we will either have to always notify users they may need to re-setup an action connector, or never show it and rely on docs to communicate this.

[ravikesarwani]

Currently we deploy new alerts when user visits the SM UI. Can we add a generic message whenever we deploy new alerts? Something on the lines of "New alerts have been created. Review the alert definition using Setup mode and configure additional action connectors to get notified via your favorite method".
The message comes only when new alerts have been created.

This will help cover many different use cases.

[chrisronline]

@ravikesarwani Sure, sounds good

[chrisronline]

@ravikesarwani Currently, these legacy cluster alerts are limited to gold+ customers, primarily because they relied on watcher which is a gold+ feature. With this change, I'm assuming we'll bring these back down to basic (along with the rest of monitoring) and just wanted to double check that with you.

[ravikesarwani]

Yes, in subscription page we will remove the following row in 7.12 (Automatic stack issue alerts). We added "Kibana alerting and actions" and has check mark in Basic.

We should account for doc changes related to this change as well.