Update Python auto-instrumentation injection

[mat-rumian]

PR can be merged only after #961

PR update Python auto-instrumentation injection because of changes in opentelemetry-distro package which sets OTEL_METRICS_EXPORTER to otlp_proto_grpc. Accordingly to the comment we have to switch it to otlp_proto_http.

[pavolloffay]

I like the list, however, what are the implications for the user? We should document what value of spec.exporter.endpoint should be set for each language.

Then if the exporter endpoint differs per language how can the CR be configured for multiple languages?

[mat-rumian]

I've added little bit about Instrumentation resource and some information about each supported auto-instrumentation. I think it will make easier for end user to run it.

[pavolloffay]

@mat-rumian could you please split the PR to python specific changes and readme changes?

I am not sure if you noticed but there is https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md docs for the CRs.

I would prefer to keep the readme minimal with example CRs people can apply to the cluster. I like that you ar documenting the ENV vars for fields in the CRD, but I would prefer to add this as godoc on the CRD.

[pavolloffay]

I would like to keep this as an reference example people can apply to the cluster

[yuriolisa]

LGTM.

[ajurge]

I have tried setting the OTEL_METRICS_EXPORTER=otlp_proto_http but i do get the following error below. Is this PR going to fix this issue and we should wait for it to be merged or is there a workaround using environmental variables?

Configuration of configurator failed
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 273, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 289, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 233, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 201, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 170, in _import_config_components
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_http' not found in entry points for 'opentelemetry_metrics_exporter'
Failed to auto initialize opentelemetry
Traceback (most recent call last):
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 121, in initialize
    _load_configurators()
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 109, in _load_configurators
    raise exc
  File "/otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation/sitecustomize.py", line 105, in _load_configurators
    entry_point.load()().configure(auto_instrumentation_version=__version__)  # type: ignore
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 273, in configure
    self._configure(**kwargs)
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 289, in _configure
    _initialize_components(kwargs.get("auto_instrumentation_version"))
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 233, in _initialize_components
    trace_exporters, metric_exporters, log_exporters = _import_exporters(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 201, in _import_exporters
    for (exporter_name, exporter_impl,) in _import_config_components(
  File "/otel-auto-instrumentation/opentelemetry/sdk/_configuration/__init__.py", line 170, in _import_config_components
    raise RuntimeError(
RuntimeError: Requested component 'otlp_proto_http' not found in entry points for 'opentelemetry_metrics_exporter'
 * Serving Flask app 'server' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.

Below are the instrumentation and deployment manifests. We use autoinstrumentation-python version 0.32b0.

Instrumentation:

---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: otel-agent
  namespace: default
spec:
  exporter:
    endpoint: http://otel-collector.observability:4318
  propagators:
  - tracecontext
  - xray
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.32b0

Manifest:

spec:
  automountServiceAccountToken: false
  containers:
    - args:
        - src/server.py
      command:
        - python
      env:
        - name: OTEL_METRICS_EXPORTER
          value: otlp_proto_http
        - name: OTEL_SERVICE_NAME
          value: example-python
        - name: PYTHONPATH
          value: >-
            /otel-auto-instrumentation/opentelemetry/instrumentation/auto_instrumentation:/otel-auto-instrumentation
        - name: OTEL_TRACES_EXPORTER
          value: otlp_proto_http
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: 'http://otel-collector.observability:4318'
        - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: OTEL_PROPAGATORS
          value: 'tracecontext,xray'
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: >-
            k8s.container.name=app,k8s.deployment.name=example-python,k8s.namespace.name=default,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.replicaset.name=example-python-959676bdf

[ianmcnally]

can this be merged soon? there are two other diffs (#1012 which is mine, and #1001) which update the python auto instrumentation version so that it mirrors the current release

[mat-rumian]

Hello Everyone,
first of all I'm sorry for late response. I found that metrics are not supported by otlp_proto_http exporter. Only gRPC exporter supports both traces and metrics. I'm not sure what we should do about it - disable metrics configuration by default because of this or include grpc exporter for our Python 3.10 docker img.

@pavolloffay I've removed almost all the changes made in README. I will prepare separated PR for them.

[pavolloffay]

We don't use the distro[otlp] option which automatically includes exporters since gRPC is not appropriate for
injected auto-instrumentation, where it has a strict dependency on the OS / Python version the artifact is built for.

Do you have a link that explains the above for OTLP gRPC exporter? Or a link that lists what OS or python versions are supported?

[mat-rumian]

Do you have a link that explains the above for OTLP gRPC exporter? Or a link that lists what OS or python versions are supported?

I've never seen anything about supported OSs beside OTLP gRPC exporter description

[pavolloffay]

I don't see any OS dependencies, however, I do see python versions mentioned there https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-otlp-proto-grpc/setup.cfg#L29-L35. Does it mean that only python version matters?

[ajurge]

Hi, is there any update on python auto-instrumentation ?

[mat-rumian]

I'm sorry I couldn't find a time to solve that on time. Closing as outdated (newer instrumentation already supported).