multiple OpenTelemetry Instrumentation doesn't work

[elvis-cai]

we are using java auto injection and it works as expected, and recently we added another nodejs Instrumentation kind, after we added it in the same ns, no pods could be created, from the kube events it's reporting

error: 15s         Warning   FailedCreate             replicaset/*my-dash*-678b655b54                      Error creating: admission webhook “mpod.kb.io” denied the request: multiple OpenTelemetry Instrumentation instances available, cannot determine which one to select

also from the controller, it's reporting

{“level”:”error”,”ts”:1644297280.4552245,”msg”:”failed to select an OpenTelemetry Instrumentation instance for this pod”,”namespace”:””,”name”:””,”error”:”multiple OpenTelemetry Instrumentation instances available, cannot determine which one to select”,”stacktrace”:”github.com/open-telemetry/opentelemetry-operator/internal/webhookhandler.(*podSidecarInjector).Handle\n\t/workspace/internal/webhookhandler/webhookhandler.go:85\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/webhook/admission/webhook.go:146\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/webhook/admission/http.go:99\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:40\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:101\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2425\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2879\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1930”}

this is the yaml file we are using

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: splunk-java-instrumentation
  namespace: test
spec:
  exporter:
    endpoint: "http://$(SPLUNK_OTEL_AGENT):4317"
  java:
    image: asia.gcr.io/mycompany/autoinstrumentation-java:v1.7.3
  resource:
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"
---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: splunk-node-jsinstrumentation
  namespace: test
spec:
  exporter:
    endpoint: "http://$(SPLUNK_OTEL_AGENT):4317"
  nodejs:
    image: asia.gcr.io/mycompany/autoinstrumentation-nodejs:0.27.0
  resource:
  propagators:
    - tracecontext
    - baggage
    - b3
  sampler:
    type: parentbased_traceidratio
    argument: "0.25"

Also had a quick search in issue list, just wondering is it related to #529? so we can selectly deploy different lang applications or same lang application with different Instrumentation configs.

[pavolloffay]

How are you assigning instrumentation to a pod? Instead of using "true" as the value just specify the instrumentation name and it should work.

[elvis-cai]

Thanks @pavolloffay , using named instrumentation works 👍
also notised when using namesapce annotation for java, if there's another app in the same ns using nodejs, I need explicitly set instrumentation.opentelemetry.io/inject-java: "false" and instrumentation.opentelemetry.io/inject-nodejs: "my-instrumentation" to make it work, otherwise, java and nodejs will be injected at the same time.

[pavolloffay]

also notised when using namesapce annotation for java, if there's another app in the same ns using nodejs, I need explicitly set instrumentation.opentelemetry.io/inject-java: "false" and instrumentation.opentelemetry.io/inject-nodejs: "my-instrumentation"

That is expected, instrumentation.opentelemetry.io/inject-java applied to a namespace enables injection to all workloads in that namespace.

[pavolloffay]

I am closing this as it seems like resolved. Please re-open if there is a proposal for change or improvement.