[Change Proposal] Support additional asset types in content packages

[kpollich]

Follow-up for #351

Today, content packages only supports dashboards or simple Kibana saved objects that can be directly imported. We should expand this support to allow for all Kibana asset types listed here: https://github.com/elastic/package-spec?tab=readme-ov-file#supported-assets

[kpollich]

@jsoriano Now that content packages are GA, it'd be great to provide a path to get the prebuilt rules package moved over to type: content. I think we need to add security rules as a supported asset type and then things should work pretty much as expected? Not sure if I'm missing any complexity there. I'm going to schedule this for our next sprint to take a look.

[jsoriano]

Yes, in principle we can start adding additional assets. For prebuilt rules I think we also needed to improve support for big files and packages in Fleet.

[mrodm]

Currently, the Kibana assets that are allowed in content packages are dashboards and SLOs (kibana folder spec definition)

The current list of Kibana assets that are supported in the integration packages is defined here:

• Dashboards (dashboard folder).
• Visualizations (visualization folder).
• Searches (search folder).
• Maps (map folder).
• Lens (lens folder).
• Index Patterns (index_pattern folder).
• Security Rules (security_rule folder).
• CSP Rule Templates (csp_rule_template).
• ML modules (ml_module folder).
• Tags (tag folder).
• Osquery Pack Assets (osquery_pack_asset folder).
• Osquery Saved Queries (osquery_saved_query folder).
• SLOs (slo folder).

As it is recommended to define visualizations, maps, lens, ... assets inside the same dashboards, probably those kind of assets could be skipped (related to #316).
Currently, the following Kibana assets cannot be defined via references in dashboards (package-spec validation rule):

• Lens
• Maps
• Searches
• Visualizations

Looking into the packages defined in the integrations repository, there are some assets that it looks like they would require transforms or data stream definitions (or at least those packages contain those other resources). Examples found:

• osquery_pack_assets and osquery_saved-queries (require data stream definitions?) https://github.com/elastic/integrations/tree/bcb031edf95b979a50749ba834a50fa8a79bf355/packages/osquery_manager
• ml_modules (require tansforms?) https://github.com/elastic/integrations/tree/bcb031edf95b979a50749ba834a50fa8a79bf355/packages/ded
• ml_modules (require tansforms?) https://github.com/elastic/integrations/tree/bcb031edf95b979a50749ba834a50fa8a79bf355/packages/dga
• csp_rule_templates (require data stream definitions?) https://github.com/elastic/integrations/tree/bcb031edf95b979a50749ba834a50fa8a79bf355/packages/cloud_security_posture

Taking that into account, what would be the Kibana assets that we should add support now in content packages? Maybe start adding just Security Rules (e.g. used in security_detection_engine)? @jsoriano @kpollich

If possible, I think it would be better to add support for new assets in content packages when they are required (on demand).

[mrodm]

Here in this PR #885, it is being added the support to include security_rule assets in content packages.

Is there any other Kibana asset type that should be included (taking into account #803 (comment))?

[mrodm]

Looking to the usage of other Kibana asset types, there are two other assets that probably they could be added to the content packages too: Index Patterns and Tags.

There are some packages using those assets specifically:

• Index Patterns:

 $ find packages -type d -name index_pattern
packages/elasticsearch/kibana/index_pattern
packages/logstash/kibana/index_pattern
packages/network_traffic/kibana/index_pattern
packages/cloud_defend/kibana/index_pattern
packages/cloud_asset_inventory/kibana/index_pattern
packages/beat/kibana/index_pattern

• Tags:

 $ find packages -type d -name tag | wc -l
51

These assets can be referenced from dashboards. They are not part of the assets in the validation rule mentioned above.

Regarding tag assets, there are some packages in the integrations repo that define the tags created by fleet, but others like ti_eset create their own tags:

• ti_eset defined its own tag: https://github.com/elastic/integrations/blob/ea64e9b6445d61c7287e69d4aae8587f80b81fbf/packages/ti_eset/kibana/tag/ti_eset-ed13a500-8192-11ee-a704-a3364df0a052.json
• bitdefender defined tags created by Fleet (managed tags): https://github.com/elastic/integrations/blob/ea64e9b6445d61c7287e69d4aae8587f80b81fbf/packages/bitdefender/kibana/tag/bitdefender-fleet-managed-default.json

@kpollich @jsoriano do you think it would be good to add those two assets to the content packages too? Or should we just add security_rule assets for now?

[mrodm]

Talked to @jsoriano offline about this and we decided to just add Security Rule Kibana assets for now.

[mrodm]

Closing after being merged #885