Infrastructure inventory UI is broken due to missing mapping for `system.process.cpu.start_time`

[lahsivjar]

APM Server version (apm-server version): >= 8.15.5

Description of the problem including expected versus actual behavior: In #13196 we fixed a long-standing bug in APM server where it didn't support the infrastructure UI in Kibana for OpenTelemetry collector running with the hostmetrics receiver. The feature, as expected, is still live however the processes tab for the discovered hosts is now blank (with no data).

This is due to disabling date detection on APM mappings done via elastic/elasticsearch#116995 and released to 8.15.5 and 8.16.0 (to be fair, 8.16.0 already had the date detection disabled due to update in the base mappings done via https://github.com/elastic/elasticsearch/pull/111091/files PR). Without date_detection enabled, the field system.cpu.process.start_time which is added as an enrichment via the opentelemetry-lib is not indexed as a field. The Kibana UI expects this field to be indexed as a date and uses its string representation to parse the date back to a date object (ref). A temporary fix for this could be to have a date mapping explicitly set for metrics custom component template:

PUT _component_template/metrics-apm.app@custom
{
  "template": {
    "mappings": {
      "properties": {
        "system.process.cpu.start_time": {
          "type": "date"
        }
      }
    }
  }
}

In addition to this, there is another issue that prevents the hostmetrics UI to be populated. https://github.com/elastic/kibana/pull/209076/files PR changed the command line attribute used in the processes tab from system.process.cmdline to process.command_line, however (as I painfully discovered during a previous debug session), this would break the UI as there will be docs where process.command_lineattribute is present but attributes like user.name, process.pid, etc are not present causing the aggregation for the processes list to be broken. IIANM, this issue should also happen with the OTel native ingestion path involving hostmetrics.

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including server configuration, agent(s) used, etc. The easier you make it
for us to reproduce it, the more likely that somebody will take the time to
look at it.

1. Create a deployment with 8.16.0++ version of the stack
2. Run the opentelemetry collector with hostmetrice receiver configured (attached an OTel config file below).
3. Open Infrastructure/Hosts UI, select a host, and open the Processes tab for the host. Expectation is that this should be populated.

Provide logs (if relevant): N/A

[simitt]

@akhileshpok for input on priority of fixing this bug in the apm-server.

In addition to this, there is another issue that prevents the hostmetrics UI to be populated. https://github.com/elastic/kibana/pull/209076/files PR changed the command line attribute used in the processes tab from system.process.cmdline to process.command_line, however (as I painfully discovered during a previous debug session), this would break the UI as there will be docs where process.command_lineattribute is present but attributes like user.name, process.pid, etc are not present causing the aggregation for the processes list to be broken. IIANM, this issue should also happen with the OTel native ingestion path involving hostmetrics.

@lahsivjar this sounds like something more complicated than adding a mapping for system.process.cpu.start_time. You also mentioned that this is likely broken for otel native ingestion. Does it make sense to address this separately or would you keep the issues together?

[smith]

The same UI is broken when using EDOT without APM server:

• elasticinframetricsprocessor sets the data_stream.dataset based on the hostmetrics scraper, to system.cpu, for example: https://github.com/smith/opentelemetry-lib/blob/1a05d8fc8a7e7e0663e11419e0e40167a87ea54e/remappers/hostmetrics/hostmetrics.go#L100
• elasticsearchexporter sets data_stream.dataset based on the receiver name to hostmetrics.otel: https://github.com/smith/opentelemetry-collector-contrib/blob/95e35ee936de87074edd0f8cf5930f3a934f336a/exporter/elasticsearchexporter/data_stream_router.go#L192
• Because of metrics_dynamic_index, docs go into metrics-hostmetrics.otel-default
• system.process.cpu.start_time is mapped as a keyword because the hostmetrics documents are not in the metrics-system.cpu-default index and the index doesn't have the mappings from the system integration

Should this be a separate issue on the exporter repo, since this path doesn't use APM?

[axw]

IIANM the EDOT bug scenario is due to open-telemetry/opentelemetry-collector-contrib#38500 (comment)

[akhileshpok]

@simitt - Can I please get an estimate for getting this issue fixed in the APM server? Thanks

[simitt]

@lahsivjar and I synced on this:
(1) date_detection being disabled: having it enabled by default caused problems in the past, however we should treat it as a bug that it was disabled for all the known date fields also and it should be fixed by enabling it explicitly. This is expected to be a straightforward fix (~1 day).
(2) hostmetrics UI issue: this has been mentioned here to make it clear that the UI won't be fixed only with the date detection fix, but actually requires a fix in the UI. So this is a noop for APM Server.

[simitt]

IMO we should tackle (1) as quickly as possible. @raultorrecilla can this task please be scheduled in the next iteration; @akhileshpok let us know if you disagree with this, IMO we should get a fix out.