[elasticsearch] Extension of the Elasticsearch integration with datastream-centric stats

[3kt]

Proposed commit message

This adds a couple new assets to the elasticsearch integration (which natively aims to monitor Elasticsearch clusters):

• A transform job that pivots index stats (either provided by this same integration, or previously shipped with Metricbeat).
• An ingest pipeline that cleans up the documents created by the transform job.
• A dashboard consuming the resulting data to provide datastream and tier centric consumption information.

This aims to provide stack-native capabilities that are otherwise fulfilled through custom scripting (known as "consumption framework" within Elastic).

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Requires multiple dependencies:

• Index stats enhancement: creation date and tier_preference elasticsearch#116339 to be merged for index creation date and data tier to be exposed in the index stats API.
• The new fields to be correctly collected by Metricbeat or the Elasticsearch monitoring integration.
• The new fields to be correctly mapped in Elasticsearch.

Screenshots

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 5b55065

Failed CI Steps

• :pipeline:⬆️ Upload Pipeline: .buildkite/pipeline.yml

History