Elastic Exporter: Elastic APM service graph "Time spent by span type" is empty

[cyrille-leclerc]

Describe the bug

Elastic APM service graph "Time spent by span type" is empty with a message "No data within this time range." when using the OpenTelemetry Java instrumentation. On the other side, when using the Elastic APM Java agent with the same application, the graph is properly filled.

Steps to reproduce

Instrument a java application connecting to a database and to a downstream service and look at the "Time spent by span type" graph.

Here is a sample with Spring Boot and PostgreSQL: https://github.com/cyrille-leclerc/my-shopping-cart/tree/otel-2020-10-15

What did you expect to see?

I expected to see the service graph "Time spent by span type" filled.

What did you see instead?

The graph "Time spent by span type" is empty with a message "No data within this time range."

What version did you use?

OpenTelemetry Collector Contrib Version: v0.12.0
OpenTelemetry Java Instrumentation Version: v0.8.0

What config did you use?

extensions:
  zpages:
    endpoint: :55679

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: '127.0.0.1:55680'
  hostmetrics:
    collection_interval: 1m
    scrapers:
      cpu:
      load:
      memory:
#      filesystem:
#      network:

processors:
  batch: null
  queued_retry: null

exporters:
  elastic:
    apm_server_url: 'http://localhost:8200'
    secret_token: 'my_secret_token'
  logging:
    loglevel: info
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    metrics:
      receivers:
        - otlp
        - hostmetrics
      exporters:
        - logging
        - prometheus
        - elastic
    traces:
      receivers:
        - otlp
      processors:
        - batch
      exporters:
        - elastic
  extensions: [zpages]

Environment
OS: MAcOS 10.15.7

Additional context
Add any other context about the problem here.

[cyrille-leclerc]

cc @axw

[axw]

Indeed, this is a known limitation. The Elastic APM agents calculate the "Time spent by span type" metrics; it is impractical to calculate them once the spans have left the agent, as it depends on knowledge of the span hierarchy. To support this feature, I believe we would need to implement language-specific span processors for each OpenTelemetry SDK.

[jkowall]

We are considering building an aggregated metric calculator pipeline in the Otel collector which we would use to export metrics off spans. There are some challenges with the current SDK in running two pipelines (exporting traces and metrics) from a single trace input, but we are considering this. It will be discussed on the Jaeger community call today.

[axw]

That sounds like a useful feature!

A span metrics pipeline might work but would have to aggregate complete traces (a la tail-based sampling), as the referenced metrics take into account time spent in nested spans. A more apt name for the metrics/graph might be "Time spent by span type, excluding any time spent in children of spans of that type."

[axw]

@cyrille-leclerc I don't see this coming any time soon. I'm inclined to close as WONTFIX and document the limitation for now, and consider implementing span processors for each of the SDKs to calculate and report the relevant metrics. WDYT?

[cyrille-leclerc]

Thanks @axw . Closing for the moment as "won't fix" sounds good to me.

[cyrille-leclerc]

Close as won't fix for the moment.

[kes2464]

Hi, is there any update on this?

[axw]

@kes2464 no, sorry. Please subscribe to elastic/apm#471 for updates.