[connector/spanmetrics] Support multiple instrumentation scopes

[fstab]

Component(s)

connector/spanmetrics

Is your feature request related to a problem? Please describe.

OpenTelemetry has the concept of instrumentation scopes. In most cases, the instrumentation scope is the name and version of the Instrumentation Library.

Here's an example where this is particularly useful: We have an eBPF-based auto-instrumentation tool that produces HTTP spans. However, if an application is already instrumented with an OTel SDK on the on the application level we get duplicate spans: Spans from the OTel SDK and spans from the eBPF tool. Thanks to instrumentation scopes that is not an issue: The spans will have different scopes, and we can use that in the monitoring backend to distinguish these spans.

Now, the spanmetrics connector does not keep the original instrumentation scope, but replaces it with a hard-coded scope name spanmetricsconnector. As a result, spans with different scopes are collapsed into the same spanmetrics leading to incorrect results when multiple libraries / tools instrument the same calls.

I think this is the line where the hard-coded spanmetricsconnector scope is coming from: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/connector/spanmetricsconnector/connector.go#L257-L258

Describe the solution you'd like

Keep the original instrumentation scope when producing span metrics.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• connector/spanmetrics: @albertteoh @portertech

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @fstab, it looks like this was an intentional design decision to show that the metrics are being generated from incoming spans in the connector. Context in this issue and the PR for it.

Would it work for you to use the dimensions or namespace config options to differentiate metrics from each other?

[fstab]

Thanks @crobert-1. I don't see how to solve this with dimensions, maybe I'm missing something.

• The triplet service.namespace,service.name,service.instance.id uniquely identifies an instance of a service. If we choose different values for any of these, then it has to be interpreted as multiple instances, not as a single instance with multiple instrumentation scopes.
• dimensions will not differ if the same calls are instrumented with multiple instrumentation scopes. Whatever span attribute we select should have the same value for all instrumentation scopes.

The only way I can see to differentiate multiple instrumentation scopes is to produce spanmetrics with different instrumentation scopes when the incoming spans have different instrumentation scopes.

I also think this issue will become increasingly relevant over time. Maybe today having an eBPF-based instrumentation for a service that's also instrumented with an SDK is an uncommon scenario. However, more and more application frameworks are adding built-in OpenTelemetry support (like Java's Spring Boot for example), and thus duplications of built-in instrumentation and auto-instrumentation will become more common. As I understand it instrumentation scope is intended to differentiate between multiple instrumentations in these cases.

[fstab]

An alternative would be to prefix the original instrumentation scope name, something like

"spanmetrics/" + <orig scope name>

That way we would not reuse the original scope, but still produce different scopes if there are multiple different original scopes.

[gouthamve]

Hi I opened the issue about resource attributes and I didn't give scopes much thought tbh. I am not sure setting the scope name to spanmetrics makes much sense. The typical usage of the metrics involves setting the namespace to something like span.metrics, so that the metrics produced are span_metrics_calls_total, etc.

I think the namespace is much better place to indicate where the metrics are coming from rather than using use OTel scopes.

[Frapschen]

As you said

Thanks to instrumentation scopes that is not an issue: The spans will have different scopes, and we can use that in the monitoring backend to distinguish these spans

The key problem is that the spanmetrics processor cannot directly access span instrumentation scopes.

Have you tried the transformprocessor? According to its doc, you can modify telemetry based on a configuration like this:

transform:
  error_mode: ignore
  trace_statements:
    - context: span
      statements:
        - set(attributes["scope.name"], instrumentation_scope.name)
        - set(attributes["scope.version"], instrumentation_scope.version)

then you can add the two span attributes into spanmetrics's dimensions

@fstab hope it can help you.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[sirianni]

The key problem is that the spanmetrics processor cannot directly access span instrumentation scopes.

Isn't that the issue to fix?

The spanmetrics connector should expose the name and version fields from the scope as elements that can be set in the dimensions configuration.

This is how Datadog trace stats work

trace.go.opentelemetry.io_contrib_instrumentation_github.com_gorilla_mux_otelmux.server.hits
trace.go.opentelemetry.io_contrib_instrumentation_net_http_otelhttp.client.hits
trace.io.opentelemetry.jetty_11.0.server.hts

Needing to use the transform processor is a workaround. The spanmetrics processor should support this directly.