[Stack Monitoring] [Test Scenario] Out of the box alerting

[chrisronline]

Summary

Stack Monitoring provides a set of out-of-the-box alerts, created by simply loading the Stack Monitoring UI within Kibana. The default action for each alert is a server log and the action messaging is controlled by the Stack Monitoring UI code directly.

PRs

Original, and CPU alert: #68805
Disk usage alert: #75419
JVM memory usage alert: #79039
Missing monitoring data alert: #78208
Threadpool rejections alert: #79433

Testing

Creation

• Ensure alerts are created once visiting the Stack Monitoring UI
• Ensure a user with the minimum set of monitoring permissions is able to create and manage alerts

Management

• Ensure that you can view and manage these alerts when Setup Mode is active
• Ensure you can properly add an additional action (to any alert) and it works as expected

UX

• Ensure you can see all created alerts while in Setup Mode
• Ensure you can properly navigate between display modes of alerts [Monitoring] Some progress on making alerts better in the UI #81569

Specific alerts

• Ensure you can properly trigger and see the server log for the CPU usage alert
• Ensure you can properly trigger and see the server log for the disk usage alert
• Ensure you can properly trigger and see the server log for the jvm memory alert
• Ensure you can properly trigger and see the server log for the missing monitoring data alert (Note: This alert is only concerned with Elasticsearch now and no longer looks at other stack products [Monitoring] Missing monitoring data alert firing for version upgrade and configuration changes for Kibana in Cloud #83309)
• Ensure you can properly trigger and see the server log for both threadpool rejection alerts
• Ensure you can properly trigger and see the server log for the legacy cluster health alert
• Ensure you can properly trigger and see the server log for the legacy nodes change alert
• Ensure you can properly trigger and see the server log for the legacy Elasticsearch version mismatch alert
• Ensure you can properly trigger and see the server log for the legacy Kibana version mismatch alert
• Ensure you can properly trigger and see the server log for the legacy Logstash version mismatch alert
• Ensure you can properly trigger and see the server log for the legacy license expiration alert

Edge cases

• Ensure a CCS environment works as expected
• Ensure Setup Mode works as expected on cloud ([Monitoring] Ensure setup mode works on cloud but only for alerts #73127)
• Ensure a dedicated monitoring cluster environment works as expected
• Ensure a quality UX with multiple Kibanas (we recommend a dedicated Kibana for a dedicated monitoring cluster) ("Duplicate" alerts can potentially exist based on what the user is doing in each Kibana but they should be able to easily disable the ones they don't want)
• Ensure Stack Monitoring UI works properly when alerts are not available (both because ssl is disabled and an encryption key is not set: [Monitoring] Fix UI error when alerting is not available #77179)
• Ensure Stack Monitoring alerts are not editable or createable in the Alerts & Management UI ([Monitoring] Prevent edit/create for Stack Monitoring alerts in Alerts Management #77097)

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[Zacqary]

Ensure you can properly navigate between display modes of alerts

Not quite sure how to test this one. Does this state only happen if you're monitoring multiple clusters?

[chrisronline]

@Zacqary

Sure! In #81569, we introduce two ways to view alerts in the UI - grouped by "node/instance" or grouped by the alert type. That line item means we should just verify you can toggle between the two and both of them make sense and the grouping looks right

[Zacqary]

@chrisronline Got it, is it expected that the toggle switch doesn't appear if you're only monitoring a single node?

[chrisronline]

@Zacqary Negative. That's something we should probably optimize in the near future, but it does not do that now.

[Zacqary]

Okay, well I'm not seeing the Group by node toggle at all. Guess I can't pass that test case then?

[chrisronline]

@Zacqary Oh wow, that's weird. Do you mind posting a screenshot of what you are seeing?

[Zacqary]

Just not seeing Group By Node anywhere. This example is with only one node, but I still wasn't able to see it when adding more nodes.

[Zacqary]

Ensure you can properly trigger and see the server log for both threadpool rejection alerts

Which ones are these? Never mind, missed em in the Errors and Exceptions section

[chrisronline]

@Zacqary I'm a bit dumbfounded. #85719 made it into 7.11, as you can see here but I don't know why it's not showing up in the UI. I just tested on cloud staging, but I'm going to test locally and see if I can figure out what's up

[Zacqary]

For the threadpool alerts, not sure if I'm doing it right but it doesn't seem to be firing when I:

• Set both Write and Read to alert every 1 minute, and fire on 1 rejection
• Do a GET .monitoring-es*/_search to find the most recent document with a node_stats field
• POST .monitoring-es*/_update/<doc id> to set node_stats.thread_pool.search.rejected to a number greater than 0, same for write.rejected