Elastic Exporter: Otel Collector appears as the service "unknown" in Elastic APM (services inventory & service map)

[cyrille-leclerc]

Describe the bug

When reporting metrics from the collector (e.g. hostmetrics), the OpenTelemetry Collector appears in Elastic APM as "unknown" with zero transaction reported.

Steps to reproduce

Monitor an app with the opentelemetry-java-instrumentation and configure OpenTelemetry Collector to collect hostmetrics.

What did you expect to see?

I expected the Elastic APM Service inventory to not report the collector, especially if it is named "unknown" and I didn't expect to see the collector on the Elastic APM Service Map.

An alternative behaviour would be to see it with a meaningful name, e.g. "OpenTelemetry Collector"

What did you see instead?

A service "unknown" as showed in the screenshots above

What version did you use?
OpenTelemetry Collector Contrib Version: 0.12.0
OpenTelemetry Java Instrumentation Version: 0.8.0

What config did you use?

extensions:
  zpages:
    endpoint: :55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: '127.0.0.1:55680'
  hostmetrics:
    collection_interval: 1m
    scrapers:
      cpu:
      load:
      memory:
#      filesystem:
#      network:

processors:
  batch: null
  queued_retry: null

exporters:
  elastic:
    apm_server_url: 'http://localhost:8200'
    secret_token: 'my_secret_token'
  logging:
    loglevel: info
  jaeger:
    endpoint: 'localhost:14250'
    insecure: true
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers:
        - otlp
        - hostmetrics
      exporters:
        - logging
        - prometheus
        - elastic
    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - elastic
#        - jaeger
#        - logging
  extensions: [zpages]

Environment
OS: MAcOS 10.15.7

Additional context
Add any other context about the problem here.

[cyrille-leclerc]

cc @axw

[axw]

Yeah, this isn't great. This is an artefact of passing the metrics through the APM Server's API, which essentially treats all metrics as application (service) metrics. Thus, the presence of the metrics manifests a service in APM.

Really they're infrastructure metrics, and should not have a service.name associated. I don't think we can reasonably solve this until/unless we ingest OTLP directly in APM Server. At that point we can more reasonably differentiate infrastructure metrics from application metrics, and only associate the latter with a service name.

[axw]

@cyrille-leclerc as mentioned above, I don't think we can fix this with the current exporter approach. We would instead need to ingest OTLP in APM Server, or otherwise bypass APM Server altogether. I suggest we close and document the limitation for now.

[cyrille-leclerc]

Close as won't fix for the moment.

[amccool]

this is still problem. hostmetrics needs a service.name (like infrastructure)