All metrics scraped from push gateway have same job label

[kpanic9]

Component(s)

receiver/prometheus

What happened?

Description

In our setup, we have a push gateway which jobs push metrics to. Each of the metrics pushed by the jobs have a different job label. We want to preserve the job label set by the metric publishing job. But when we set honor_labels: true in the otel prometheus receiver configuration for the scrape job, all metrics scraped from the push gateway has a single value for the job label. The value set for the job label is from a one set of metrics pushed by a job.

Steps to Reproduce

Configure a push gateway, push few metrics to it with different values for job label.
Configure OTEL collector to scrape push gateway.
Check the values for the job label.

Expected Result

Metrics scraped from push gateway should have the job label value set by the metrics publisher.

Actual Result

All metrics scraped from push gateway has a single value for the job label.

Collector version

v0.103.0

Environment information

No response

OpenTelemetry Collector configuration

receivers:
      prometheus/2:
        config:
          scrape_configs:
          - honor_labels: true
            job_name: app-platform-pushgateway
            kubernetes_sd_configs:
            - namespaces:
                names:
                - app-platform-monitoring
              role: pod
            relabel_configs:
            - action: keep
              regex: Running
              source_labels:
              - __meta_kubernetes_pod_phase
            - action: keep
              regex: true
              source_labels:
              - __meta_kubernetes_pod_ready
            - action: keep
              regex: metrics
              source_labels:
              - __meta_kubernetes_pod_container_port_name
            scheme: http
            scrape_interval: 30s
            scrape_timeout: 10s

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bacherfl]

I was looking into this to get a better understanding of this receiver, and also reproduced this behavior. Looking at the code, it seems like after every scrape for a certain scrape config, all gathered metrics are put into the same resource, which is created in the initTransaction method: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/internal/transaction.go#L358 - the name of the resource will be set based on the job label - which is the name of the scrape config if honor_labels is set to false, but with honor_labels set to true it can happen that we have multiple different values for that label.
When attaching the different data points of metrics with the same name, e.g. the job and instance labels are among the labels which will not be added to the datapoint attributes (see.

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Line 52 in 76c03ec

func getSortedNotUsefulLabels(mType pmetric.MetricType) []string {

) - so from my understanding this could be an explanation for why all different instances of a metric are aggregated under the same job label.

Question to the code owners - @Aneurysm9 @dashpole - is the creation of one resource per transaction intended, or does the logic need to be adjusted to account for the possibility of having different values for the job label in case honor_labels is set to true, i.e. should in this case multiple resources be created in the same transaction, based on the set of different values for the job label?

[dashpole]

Good find. The logic needs to be adjusted to account for multiple resources. We should create a new, unique resource for each combination of job + instance.

[bacherfl]

Good find. The logic needs to be adjusted to account for multiple resources. We should create a new, unique resource for each combination of job + instance.

Thanks for the response @dashpole - I would be happy to work on a PR for this. I already have a PoC implementation that should fix this which needs some polishing and tests, but I should have a PR ready this week

[kpanic9]

Tested the fix, it works. Thank you!