Related to: #101013

Summary

As part of #101013 we implemented simple rule execution logging to .kibana-event-log-* indices via the event_log plugin. There is still a lot to do from the tech standpoint, and this ticket tries to capture all the outstanding technical issues.

To do

New data model

• [Security Solution][Detections] New data model for Rule Execution Log and changes in HTTP APIs #119603
  • Clean HTTP API, legacy rule status SO under the hood.
• [Security Solution][Detections] New Saved Object for Rule Execution Log #119597
  • Clean HTTP API, new Rule Execution Info SO under the hood.

Stability and performance

• [Security Solution][Detections] Add error handling to Rule Execution Log #119595
• [Security Solution][Detections] Stop updating rule status on enable/disable #119596
• [Security Solution] Improve RuleExecutionLog performance #118511
• Graceful error handling when fetching rule statuses in the rules/_find endpoint (comment)
• Revisit the problem of deterministic ordering (comment) (PR)

Features

• [Alerting] Storing custom searchable rule execution data inside the rule #112193
  • will allow us to implement filtering, sorting and searching by rule execution status and metrics fields in the Rule Monitoring table
• [Security Solution] Extend event_log plugin with functionality required for Rule Execution Log #106347
• [RAC][Rule Registry] Add unique ID field for identifying all alerts created during a specific Rule execution #110135
• Add support for writing "generic" rule execution events to Event Log. They could be used to write generic errors, warnings, info and debug messages to the log. (PR)
• Log errors from the Detection Engine's "big loop" as separate events (generic errors). (comment 1, comment 2)
• [Security Solution] Rules don't always write a status when finished executing #116358

Misc

• Encapsulate the current space id in the instance of IRuleExecutionLogClient. Remove it from parameters of its methods. (PR)
• Implement a Rule Execution Logger scoped to a particular rule instance. (comment)
  • It would scope common parameters like rule itself, space id etc, and would provide a cleaner and simpler API for rule executors (PR)
  • It would be writing Kibana logs to the system logger in addition to logging execution events to Event Log. Rule executors would be using it instead of the system logger directly. (PR)
• Add test coverage.
• Improve Event Log's index bootstrapping logic to simplify development. Currently, in dev mode, if you update mappings of .kibana-event-log* indices, you won't easily get these updates in ES. If we look at x-pack/plugins/event_log/server/es/init.ts, seems like in order to propagate changes in the event log mappings to ES, we need to at least delete the index template, and then either delete the concrete indices or do a manual rollover after KIbana startup. Probably this could be solved via a "force upgrade index" setting of the event_log plugin that would be set to true locally.