k8sattributes: labels not attached to metrics from instrumented java process

[marcoboi]

Component(s)

processor/k8sattributes

Describe the issue you're reporting

Setup
I have an Otel-instrumented application sending metrics to an Otel Collector instance.
I've set up k8sattributes processor to enrich the metrics.
Metrics are then pushed into prometheus and consumed in Grafana.

Problem framing
I'm expecting the metrics from the instrumented application to carry the extra labels provided by k8sattributes.
The metrics i consume from Grafana do not have any k8s-related attributes attached to them, for instance:

process_runtime_jvm_memory_limit_bytes{
  instance="simple-java-app.simple-java-app-2-58fc45bbd7-mckvm.simple-java-app-2",
  job="simple-java-app-2",
  pool="CodeHeap 'non-profiled nmethods'",
  type="non_heap"
}

Configuration

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: "agent-collector"
spec:
  mode: daemonset
  tolerations:
    - operator: Exists
  replicas: 1
  resources:
    limits:
      memory: 500Mi
  containers:
    - env:
        - name: KUBE_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName

  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      k8sattributes:
        auth_type: serviceAccount
        passthrough: false
        filter:
          node_from_env_var: KUBE_NODE_NAME

    exporters:
      otlphttp/metrics:
        endpoint: http://{{ .Release.Name }}-prometheus-server:9090/api/v1/otlp
        tls:
          insecure: true
      debug/metrics:
        verbosity: detailed
    service:
      pipelines:
        metrics:
          receivers: [ otlp ]
          processors: [ k8sattributes ]
          exporters: [ otlphttp/metrics, debug/metrics ]

[github-actions]

Pinging code owners:

• processor/k8sattributes: @dmitryax @rmfitzpatrick @fatsheep9146 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[marcoboi]

See related discussion in this thread.

[TylerHelmuth]

Seems to be a duplicate of #34075

[marcoboi]

While not exactly a duplicate, the solution for this issue is the same as for #34075.
Namely, the attributes extracted from Otel need to be re-mapped to a format that can be understood by Prometheus.

This can be done using the following processor block in the Otel Collector pipeline:

      transform:
        metric_statements:
          - context: datapoint
            statements:
              - set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
              - set(attributes["container"], resource.attributes["k8s.container.name"])
              - set(attributes["pod"], resource.attributes["k8s.pod.name"])
              - set(attributes["pod_id"], resource.attributes["k8s.pod.uid"])
              - set(attributes["node"], resource.attributes["k8s.node.name"])
              - set(attributes["app"], resource.attributes["k8s-app"])

You can check #34075 for more details.