Summary

The migration to AAD in 8.0 highlighted several shortcomings in our ability to reliably test backwards compatibility in the Security Solution. A large part of that problem stems from the fact that we have a severe lack of data fixtures used by our automated tests that are based on reproducible, representative data from historical versions. This issue will be used to track progress towards the goal of improving this situation.

• Create a tool to create a cloud deployment, generate alerts, and capture data as fixtures
  • Also dump metadata timestamp that can be used to construct relative times for tests
• Implement a process for generating fixtures at the end of each feature freeze and integrating with tests
• Generate and commit fixtures for 8.1, 8.0, 7.17

General Approach/Strategy

Modify kbn-alert-load to use for fixture generation and export to its own repository. Include a base rule definition for each type and override for each version as needed. Use ecctl with “--docker_image” parameter to spin up the desired version(s) of Kibana in the Cloud, load the source data, create the rules, and wait for alerts to be generated. Once generated, export the alerts to their own versioned files/directories. Check these fixtures into Kibana.

Open questions

What should the source data look like? How do we generate it?
Should we use alerts from the previous and current version as part of the source data?
Which rule configurations to test?
How do we modify the fixture data to construct the relative timestamp?
How many alerts? More than one page?
Cardinality of data?