[DOCS] Ability to get the list of assets and their content for a given integration package version

[lucabelluccini]

As the elastic/integration repo:

• doesn't contain the "rendered" version of the final index templates and assets installed by each integration
• it's also NOT versioned (so we cannot see the differences across different integration package releases

As the integration packages:

• contain the exact files as we have in the elastic/integration repository

When troubleshooting, a user is unable to tell:

• What are the assets (type and naming) they're expected to see installed for a given integration
• What is the content of the assets

Bonus points:

• Know also which component will be started (Filebeat, Metricbeat, etc...)?

It would be great to have a list of assets and their contents or a way to generate them.

E.g. opening https://epr.elastic.co/epr/ti_abusech/ti_abusech-1.8.0.zip (or navigate https://github.com/elastic/integrations/tree/main/packages/ti_abusech, but we only get latest as it's not tagged) gives us all those resources but we cannot tell the final naming of the index templates or the exact rendered JSON Index Template or Ingest pipeline content.
I've tried to run some commands of https://github.com/elastic/elastic-package to see if there's a way to do it but it seems not possible.
The assets are actually parsed by Kibana and generated there by the Fleet plugin.

[jlind23]

@elastic/ecosystem Any available solution as of today to easily retrieve this?
cc @kpollich

[kpollich]

We don't expose a way to view the content for all assets at once today as the payload would be quite massive. The path to each asset is available under the assets property returned from EPR, e.g. https://epr.elastic.co/package/system/1.24.3/

"assets": [
    "/package/system/1.24.3/LICENSE.txt",
    "/package/system/1.24.3/changelog.yml",
    "/package/system/1.24.3/manifest.yml",
    "/package/system/1.24.3/docs/README.md",
    "/package/system/1.24.3/img/kibana-system.png",
    "/package/system/1.24.3/img/metricbeat_system_dashboard.png",
    "/package/system/1.24.3/img/system.svg",
    "/package/system/1.24.3/data_stream/application/manifest.yml",
...

However, each of these assets is not actually served by EPR. The list here is just an instruction for how Fleet should look up the file after the .zip archive is unpacked.

What are some asset types we'd be looking to debug here? It sounds like we're looking for what ingest pipelines, index templates, etc to expect which could easily be documented as the naming scheme is predictable.

If we're looking to track down exactly what dashboards/visualizations (these are typically what a package refers to as "assets") a package ships and present their JSON content somewhere in a unified/readable way I'm afraid that really won't be reasonable. Dashboards are often hundreds or thousands of lines of JSON and their content is not generally presentable.

Looking at the ingest pipeline and index template APIs, there's not actually a way to search/filter via query params or anything along those lines. Each of these ES objects generated by Fleet will include metadata along the lines of this, though:

"_meta": {
          "package": {
            "name": "synthetics"
          },
          "managed_by": "fleet",
          "managed": true
        },

So, filtering on the package name and managed_by: fleet would be an easy way to track down all of the installed pipelines and templates for a given package. The ES API just doesn't expose a way to do that from what I can tell.

[lucabelluccini]

Hello @kpollich - I think we can focus only on "Elasticsearch" assets.

The idea would be to be able to detect if there's a difference between the Index Template(s), Ingest pipeline(s), Component Template(s) in the package vs the one already installed in Elasticsearch.

The objective is to spot problems derived from:

• users editing the built-in assets
• bugs as [Fleet] Loading Package from ES lead to incorrect cache kibana#151195 (but I agree this might be partially not true, as if the "bug" is in the actual generation of the assets from the YAML resources, the "bug" would also show up in this functionality, as the "diff" would lead to no differences)

A sort of "dry-run" / "check" button present on each Fleet Integration which tells you if the ES assets installed <> the ones which would be installed on the same integration version.

[lucabelluccini]

This is possibly pending elastic/kibana#172393 or a platform solution to have this kind of "diff"

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!