Description
In this issue, we are trying to get the rerouting and the appropriate field naming working for system Integration with OTEL datastream.
The current flow being tried is as follows:
Take an example of CPU metrics here:
Restructuring of Fields:
For restructuring, there will be usage of processors in the ingest pipeline which works only on hierarchical fields.
There is a provision to convert flat to hierarchical fields through dot expander processor.
But currently in this case the fields are mix of flat and heirarchial. Details are below:
1. Fields are not flat:
The fields should be flat as OTEL field uses subobjects: false, but currently it is a mix of flat and hierarchical
Current output:
"metrics": {
"system.memory.actual.free": 6389440512,
"system.memory.actual.used.bytes": 1801728000,
"system.memory.used.pct": 0.9095303173272087
}
Expected Output:
metrics.system.memory.actual.free: 6389440512
metrics.system.memory.actual.used.bytes: 1801728000
metrics.system.memory.used.pct: 0.9095303173272087
Rerouting:
We want to reroute the data from metrics-generic-default to metrics-system.*-default
Currently, I am taking the rerouting decision based on this field:
`instrumentation_scope.name == "otelcol/hostmetricsreceiver/cpu".
This will bring in both the OTEL and ES metrics to the new datastream. As discussed, addition of an attribute to the event having ES documents will help in grouping only the ES metrics.
`
TBD......
Other Issues:
1. Duplication of data:
Will there be duplication, when integration upgrades to the OTEL supported Integration? The metrics will be flowing in system.cpu datastream and we will also be having metrics with the same name routed to the system.cpu dataset.
