Description
openedon Jun 9, 2020
In #18758 the decision was made to change the config structure of the Elastic Agent. This issue is to track the changes needed across all parts of the Stack. It is split up in different phases so the changes can be done in small chunks and it can be tested everything still works as expected.
Phase 1 - Remove data source
In Phase 1, the datasources array is removed from the config and all the configs on the datasource level are moved to the input level.
Before:
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
dataset: system.cpu
After:
inputs:
- namespace: default
use_output: default
type: system/metrics
streams:
- metricset: cpu
dataset: system.cpu
Changes to be made:
- Update config processing in Elastic Agent [Ingest Manager] New structure of agent configuration #19128
- Update config generation in Ingest Manager (Kibana). Datasources on saved object level should not be removed, only what is generated for the Agent is changed. [Ingest Manager] Replace
datasources
withinputs
when generating agent config kibana#69226
Phase 2 - Adjust dataset naming
Based on elastic/package-registry#482 configs related to datasets should be adjusted
Before
inputs:
- namespace: default
use_output: default
type: system/metrics
streams:
- metricset: cpu
dataset: system.cpu
After
inputs:
- dataset.namespace: default
use_output: default
type: system/metrics
streams:
- metricset: cpu
dataset.name: system.cpu
- Add support for renamed configs on the Elastic Agent side [Ingest Manager] New structure of agent configuration #19128
- Update Kibana generation of configs accordingly [Ingest Manager] Replace
datasources
withinputs
when generating agent config kibana#69226 - Adjust packages in package-storage
- Adjust package-registry and manifest
- Update integrations package generation if needed
Phase 3 - Remove datasources
The naming of datasources should be renamed in Kibana, the package-registry and the packages.
- Rename datasources (SO) in Kibana [Ingest Manager] Rename data sources to package configs kibana#70259
- Rename
datasources
list in package manifest toconfig_templates
Update package output structure package-registry#514 - Update packages in package-storage to new format
- Update integrations repositories to the new structure Update package output structure package-registry#514
- Update Kibana to support package registry changes [Ingest Manager] Support updated package output structure kibana#69864
Phase 4 - Add support for dataset.type
Currently the dataset.type is inherited from the input.type but this might not always correct. In the future it should be possible to set the dataset.type independently. Below is an example on how this could look (example does not make much sense):
inputs:
- dataset.namespace: default
use_output: default
type: system/metrics
streams:
- metricset: cpu
dataset.name: system.cpu
dataset.type: logs
- Add support for it on the Elastic Agent side [Ingest Manager] New structure of agent configuration #19128
- Adjust package-registry and manifest as needed Update package output structure package-registry#514
- Add
dataset.type
to Kibana config generation [Ingest Manager] Support updated package output structure kibana#69864