Not getting trace for all java and node js microservices

[shishir84]

Describe the bug
Not getting trace for all java and node js microservices

Steps to reproduce
i am patching the java services as below:
kubectl patch deploy/role1 -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-java": "true"}}}}}'
kubectl patch deploy/role1 -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject": "true"}}}}}'

Same way I have patched another java service. for one service i am getting complete traces as well as I can see that service in AWS Opensearch > [Observability]>[Trace analytics]>[Services]. But for another service neither the service name or trace is populated in opensearch dashboard.

What did you expect to see?
I want to see all the services which has been patched like above mentioned steps.

What did you see instead?
another service neither the service name or trace is populated in opensearch dashboard.

What version did you use?
image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.90.0 - manager
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.1 - rbac

What config did you use?
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: auto-instrumentation
spec:
exporter:
endpoint: "http://observability-collector.aws-otel-eks.svc.cluster.local:4317/"
propagators:

• tracecontext
• baggage
• b3
  sampler:
  type: parentbased_traceidratio
  argument: "0.25"
  java:
  image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  nodejs:
  image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
  image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

Environment
AWS EKS 1.27

Additional context
NodeJS:
kubectl patch deploy/management -p '{"spec": {"template": {"metadata": {"annotations": {"instrumentation.opentelemetry.io/inject-nodejs": "true"}}}}}'
kubectl patch deploy/management -p '{"spec": {"template": {"metadata": {"annotations": {"sidecar.opentelemetry.io/inject": "true"}}}}}'

[TylerHelmuth]

@shishir84 are the app being annotated and the Instrumentation CR in the same namespace? If not your annotation will not work. You'll see a log in the Operator logs saying something like "No Instrumentation resource found"

[shishir84]

Yes the microservice and the auto-instrumentation is in same namespace.

i have 20+ java based microservices. all are implemented in java 11. when i have annotated all 20+ services, i am only getting traces from 6 microservices. Not sure why is it happening. I tried this in multiple environment like dev, stage and prod. everywhere same issue.

[TylerHelmuth]

On the pods that are not working do you see the initContainer the operator injects for auto-instrumentation? You can see an example by looking at the pods that are working.

[iblancasa]

i have 20+ java based microservices. all are implemented in java 11. when i have annotated all 20+ services, i am only getting traces from 6 microservices. Not sure why is it happening. I tried this in multiple environment like dev, stage and prod. everywhere same issue.

Can you explain a little bit more about "everywhere same issue"? I mean: are there always exactly 6 applications sending traces? Are the same 6 applications?

Do you see any errors in the operator logs?