instrumentation CR spec.env not working

[Aransh]

Component(s)

auto-instrumentation

What happened?

Description

When deploying the instrumentation CR with values under spec.env, the values do not seem to propagate down/overwrite default configuration

Steps to Reproduce

Deploy the operator (in my case using opentelemetry-kube-stack helm chart version 0.6.1) with instrumentation enabled, and some values under spec.env, for example:

instrumentation:
  enabled: true
  exporter:
    endpoint: http://opentelemetry-kube-stack-cluster-stats-collector.monitoring:4317
  env:
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none

And deploy any pod using the auto-instrumentation annotation.
The goal being to disable logs and metrics instrumentation (only need traces).

Expected Result

The pod starts with env vars OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER set to "none"

Actual Result

The pods start with the default values, completely ignoring the configured env values (while the CR seems to pick up on it no problem).
To actually get this to work, must repeat these env vars under each language (very tedious and not flexible), as the "global" env section seems to have no effect, example:

instrumentation:
  enabled: true
  exporter:
    endpoint: http://opentelemetry-kube-stack-cluster-stats-collector.monitoring:4317
  java:
    env:
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none
  python:
    env:
    - name: OTEL_METRICS_EXPORTER
      value: none
    - name: OTEL_LOGS_EXPORTER
      value: none

It is possible that I am misunderstanding the use-case for this field, but I found no other clear explanation, and can't think of any other use-case except "global env - shared overwrite, language env - specific, higher precedent overwrite"

Kubernetes Version

1.31.6

Operator version

0.120.0

Collector version

0.120.0

Environment information

No response

Log output

Additional context

No response

[swiatekm]

Looks like this is indeed how it works right now. That is, the priority is:

language spec env vars > instrumentation defaults > instrumentation spec env vars

Intuitively, you're right that we would instead expect:

language spec env vars > instrumentation spec env vars > instrumentation defaults

@open-telemetry/operator-approvers do you recall if this is intentional? Git blame tells me we've had this behaviour for a long time.

[iblancasa]

I think we don't have any test for this.. so might be a bug since the thing was contributed.

[swiatekm]

We discussed this during a recent SIG meeting, and decided it's a bug we should fix. It's obviously incorrect, and considering there aren't more reports, chances are barely anyone depends on this ordering.

[Horiodino]

/assign